Skip to content

Codex updates#8

Open
BurnedChris wants to merge 7 commits intomainfrom
codex-updates
Open

Codex updates#8
BurnedChris wants to merge 7 commits intomainfrom
codex-updates

Conversation

@BurnedChris
Copy link
Copy Markdown
Contributor

No description provided.

…ookie-control'; update configurations and dependencies. Includes new METHODOLOGY.md for benchmarking guidelines and adjustments to .gitignore and package.json scripts.
…ight, Turbo, and Drizzle Kit. Refactor package.json and pnpm-lock.yaml to use catalog references for type definitions and Next.js, enhancing consistency and maintainability.
…ne benchmark configurations. Includes changes to bundle types, TypeScript settings, and enhancements to cookie banner detection across various projects.
… renaming terms for consistency. Update README.md to reflect changes in metric definitions. Remove outdated benchmark configurations and files for 'with-c15t-nextjs', 'with-c15t-react', 'with-cookie-control', 'with-cookie-yes', 'with-didomi', 'with-enzuzo', and 'with-ketch'.
…nhance clarity. Update cookie banner settings, measurement parameters, and run profiles across various benchmarks. Introduce new results schema for improved data handling and validation. Correct bundle type references in codebase to ensure consistency.
@vercel
Copy link
Copy Markdown

vercel bot commented Feb 16, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
benchmarks-with-cookie-yes Error Error Feb 16, 2026 5:27pm

Request Review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Feb 16, 2026

📝 Walkthrough

Summary by CodeRabbit

Release Notes

  • Documentation

    • Added comprehensive benchmarking methodology documentation with timing modes, metrics definitions, and scoring approach.
  • New Features

    • Expanded configuration schema with run profiles, measurement settings, and internationalization options for benchmarks.
    • Introduced dual timing measurements for cookie banners (DOM presence vs. user-visible) with visibility thresholds.
  • Chores

    • Reorganized benchmark applications with streamlined naming conventions.
    • Updated Node.js engine requirement to >=20.9.0 across projects.
    • Restructured internal packages and dependencies.

Walkthrough

This pull request restructures the monorepo by extracting benchmark runner logic from the CLI package into dedicated @consentio/benchmark, @consentio/runner, and @consentio/shared packages, and replacing the old CLI with a new cookiebench-cli package. It adds comprehensive benchmarking methodology documentation, updates benchmark configuration schemas with new run profile and measurement fields, and reorganizes benchmark app configurations across the repository.

Changes

Cohort / File(s) Summary
Benchmark Package Core
packages/benchmark/src/*, packages/benchmark/package.json, packages/benchmark/rslib.config.ts, packages/benchmark/tsconfig.json
New package exporting collectors (CookieBannerCollector, NetworkMonitor, ResourceTimingCollector, PerfumeCollector) and bundle strategy detection with comprehensive performance metrics types.
Runner Package Core
packages/runner/src/*, packages/runner/package.json, packages/runner/rslib.config.ts, packages/runner/tsconfig.json
New package providing BenchmarkRunner orchestration, PerformanceAggregator, statistical helpers, and server management utilities for executing benchmarks.
Shared Utilities Package
packages/shared/src/*, packages/shared/package.json, packages/shared/rslib.config.ts
New package centralizing reusable constants (time, size, percentage), config reading, formatting utilities, and package manager detection.
CLI Package Replacement
packages/cookiebench-cli/src/*, packages/cookiebench-cli/package.json, packages/cookiebench-cli/rslib.config.ts, packages/cookiebench-cli/tsconfig.*
New CLI package with command modules (benchmark, results, scores, save, db, migrate-results), logger, auth, config validation, and scoring utilities.
Old CLI Package Removal
packages/cli/*
Entire packages/cli directory removed including benchmark-runner, metrics-calculator, network-monitor, resource-collector, performance utilities, database commands, and results reporting.
Benchmark Configurations
benchmarks/*/config.json, benchmarks/*/package.json, benchmarks/*/next-env.d.ts
Updated all benchmark app configs with runProfile and measurement sections; renamed packages; added engines constraints; replaced version pins with catalog placeholders.
Schema Updates
packages/benchmark-schema/schema.json, packages/benchmark-schema/results.schema.json
Added new top-level schema properties (runProfile, measurement, source, includes, internationalization); introduced results schema v2 with comprehensive result structure.
Documentation
METHODOLOGY.md, README.md, .cursor/rules/ultracite.mdc, .gitignore
Added benchmarking methodology document, updated README with new timing metrics, added Ultracite AI rules, added trace directory exclusion.
Root Configuration
package.json
Updated scripts to use new Turbo-filtered workflow; dependency updates for @consentio/* packages and tool versions; removed @cookiebench/cli.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

  • Updated Deps #5 — Modifies benchmark configurations, dependency structures, and biome settings; intersects with this PR's configuration consolidation and package reorganization efforts.

Poem

🐰 Hops through the code with glee so bright,
Extractors dance—packages take flight!
Shared and runner, benchmark so keen,
The finest refactoring we've ever seen! 🎯✨

🚥 Pre-merge checks | ✅ 1 | ❌ 3

❌ Failed checks (2 warnings, 1 inconclusive)

Check name Status Explanation Resolution
Description check ⚠️ Warning No pull request description was provided by the author; this is a critical gap for understanding the context and intent of such a large changeset. Add a detailed pull request description explaining the key changes, rationale, and any breaking changes or migration steps required.
Docstring Coverage ⚠️ Warning Docstring coverage is 34.07% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The PR title 'Codex updates' is vague and does not clearly describe the primary changes in this extensive changeset. Replace with a specific title that summarizes the main change, e.g., 'Refactor CLI and create @consentio/* packages' or 'Reorganize benchmark CLI and introduce runner/benchmark packages'.
✅ Passed checks (1 passed)
Check name Status Explanation
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch codex-updates

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 59

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
benchmarks/cookie-control/app/layout.tsx (1)

15-53: 🧹 Nitpick | 🔵 Trivial

Avoid inline dangerouslySetInnerHTML to keep CSP/XSS posture.

Inline JS requires bypassing React’s safeguards and typically forces unsafe-inline CSP. Prefer next/script (or an external config file) so you can apply CSP nonces and avoid lint ignores.

🔧 Suggested refactor
-import type { Metadata } from "next";
+import type { Metadata } from "next";
+import Script from "next/script";
...
-				<script
-					src="https://cc.cdn.civiccomputing.com/9/cookieControl-9.10.1.min.js"
-					type="text/javascript"
-				/>
-				<script
-					// biome-ignore lint/security/noDangerouslySetInnerHtml: its okay to set inner html
-					dangerouslySetInnerHTML={{
-						__html: `
-							var config = {
-								apiKey: 'ce6586e299e7333a8c22432a32c1972fd859a49c',
-								product: 'COMMUNITY',
-								optionalCookies: [ ... ]
-							};
-							CookieControl.load(config);
-						`,
-					}}
-					// biome-ignore lint/security/noDangerouslySetInnerHtml: its okay to set inner html
-					type="text/javascript"
-				/>
+				<Script
+					src="https://cc.cdn.civiccomputing.com/9/cookieControl-9.10.1.min.js"
+					strategy="beforeInteractive"
+				/>
+				<Script id="cookie-control-config" strategy="beforeInteractive">
+					{`
+						var config = {
+							apiKey: 'ce6586e299e7333a8c22432a32c1972fd859a49c',
+							product: 'COMMUNITY',
+							optionalCookies: [ ... ]
+						};
+						CookieControl.load(config);
+					`}
+				</Script>
🤖 Fix all issues with AI agents
In `@benchmarks/c15t-nextjs/next-env.d.ts`:
- Line 3: Remove the manual import from the auto-generated next-env.d.ts file
(delete the line importing "./.next/types/routes.d.ts") and do not edit
next-env.d.ts again; instead add the pattern ".next/types/**/*.ts" to the
"include" array in tsconfig.json so the generated route types are picked up
globally by the TypeScript compiler.

In `@benchmarks/c15t-nextjs/package.json`:
- Line 13: Update the dependency entry for "@c15t/nextjs" in package.json from
"1.7.1" to the latest stable "1.8.3"; then regenerate your lockfile by running
your package manager (npm install or yarn install) so the updated version is
reflected in package-lock.json or yarn.lock and CI uses the new dependency.

In `@benchmarks/cookie-control/config.json`:
- Around line 18-22: The bundleType value in the techStack JSON is invalid
("iife"); update the "bundleType" property in the techStack object to the
schema-allowed value "iffe" (replace "iife" with "iffe") so the config validates
against the benchmark-schema enum for bundleType.

In `@benchmarks/cookie-yes/next-env.d.ts`:
- Around line 3-6: The import of "./.next/types/routes.d.ts" in next-env.d.ts is
auto-generated and should not be manually edited; ensure the project's Next.js
config (next.config.js/next.config.ts) enables typedRoutes: true so Next
regenerates this import consistently rather than editing next-env.d.ts
directly—open your next.config file, add or set the typedRoutes option to true,
and revert any manual changes to next-env.d.ts so the auto-generated import is
maintained by the build.

In `@benchmarks/didomi/app/layout.tsx`:
- Around line 1-6: This layout is missing the metadata export; add a server-side
export named metadata (export const metadata: Metadata = { title: "benchmark" })
but because metadata is only valid in Server Components you must remove or
relocate the "use client" usage: either drop "use client" and move the
DidomiSDK/client-specific logic into a separate client component (e.g.,
DidomiClient component) which you import into the server layout, or split this
file into a server layout that exports metadata and a companion client component
file that contains "use client" and the DidomiSDK/IDidomiObject usage; update
imports/exports accordingly to keep DidomiSDK, IDidomiObject and any
useState/useCallback calls inside the client file.
- Around line 76-85: The inline arrow handlers for onNoticeHidden and
onNoticeShown cause new function refs each render; wrap them in
React.useCallback and replace the inline props with the memoized callbacks
(e.g., create const handleNoticeHidden = useCallback(() => { if (DEBUG_DIDOMI)
console.log("Didomi notice hidden"); }, [DEBUG_DIDOMI]) and similarly
handleNoticeShown) and pass those to the DidomiSDK props to avoid unnecessary
re-renders.

In `@benchmarks/iubenda/config.json`:
- Line 25: The bundleType value in benchmarks/iubenda/config.json is using the
invalid enum "iife"; update the "bundleType" property to the allowed enum value
"iffe" (replace "iife" with "iffe") so it matches the schema used by the
benchmark-schema (valid values: "esm", "cjs", "iffe", "bundled").

In `@benchmarks/osano/app/layout.tsx`:
- Around line 17-20: The benchmark layouts differ: this file uses the Next.js
Script component with strategy="afterInteractive" (the Script import and JSX
usage), while the Enzuzo benchmark uses a raw <script> in the head; pick one
approach and make them consistent. Either replace the Script usage in this
layout with the same plain script insertion used by the Enzuzo layout, or update
the Enzuzo layout to use the Next.js Script component with the same strategy;
ensure you update the component import/usage (Script) or the head script
insertion accordingly so both benchmarks load the third‑party Osano/Enzuzo
scripts identically for comparable measurements.

In `@package.json`:
- Around line 28-30: Update the root package.json "engines" entry so the "node"
field matches the stricter workspace requirement (set to ">=20.9.0"); locate the
"engines" object and change the "node" value, ensuring consistency with
workspace packages such as packages/benchmark/package.json and
benchmarks/enzuzo/package.json to avoid engine mismatches across the monorepo.

In `@packages/benchmark-schema/results.schema.json`:
- Around line 38-46: The timestamp properties generatedAtUtc, runStartedAtUtc,
and runCompletedAtUtc are declared as "type": "string" but should enforce ISO
8601; update each property in the schema to include "format": "date-time"
alongside "type": "string" (e.g., generatedAtUtc, runStartedAtUtc,
runCompletedAtUtc) so JSON Schema validators will validate them as
RFC3339/ISO-8601 date-time strings.
- Around line 16-18: The "results" array in results.schema.json currently has no
"items" schema, allowing any content; update the "results" definition to include
an "items" schema that describes the expected element shape (e.g., an object
with required properties, types, and any nested schemas), referencing the
"results" property name so validators and docs enforce the expected result
structure (adjust property names and required fields to match your
domain-specific result object).

In `@packages/benchmark/README.md`:
- Line 104: The README's BUNDLE_TYPES lists "IIFE" while the code/schema uses
"iffe" — make them consistent by choosing one canonical value and updating the
other; locate the BUNDLE_TYPES constant and the schema validation (look for
symbols named BUNDLE_TYPES and the schema or enum that contains "iffe") and
either change the schema enum to "IIFE" or change the README/constant to "iffe"
(and update any serialization/deserialization logic that maps between
human-facing names and stored values) so both documentation and runtime
validation use the same token.
- Line 14: The README mentions "IIFE" but the benchmark schema's enum for
bundleType uses the value "iffe"; update either the schema or the docs to be
consistent: either change the enum in the benchmark-schema package to "iife" or
explicitly document the actual enum value ("iffe") in
packages/benchmark/README.md; reference the bundleType enum in the
cookiebench/benchmark-schema package (valid values: "esm", "cjs", "iffe",
"bundled") when making the change so users won't be confused by the IIFE vs iffe
spelling.

In `@packages/benchmark/src/constants.ts`:
- Around line 76-82: The BUNDLE_TYPES constant contains an incorrect literal
"iife" which doesn't match the schema (schema expects "iffe"); update the
BUNDLE_TYPES export so the bundle type matches the schema by replacing the IIFE
entry with the correct value/key (e.g., change the key to IFFE and its value to
"iffe" or at minimum change the value to "iffe"), and then update all usages
that reference BUNDLE_TYPES.IIFE to use BUNDLE_TYPES.IFFE (or the new value) so
runtime/config validation passes.

In `@packages/benchmark/src/network-monitor.ts`:
- Around line 200-206: getNetworkRequests and getMetrics expose internal mutable
state by returning direct references; change them to return defensive copies
instead (e.g., return a shallow copy of the array for getNetworkRequests and a
shallow-cloned object for getMetrics). Update the implementations of
getNetworkRequests() and getMetrics() to return copies (e.g., using array spread
or slice for networkRequests and object spread or a shallow
clone/structuredClone for metrics) so external callers cannot mutate
this.networkRequests or this.metrics; if metrics contains nested objects and
mutation protection is required, perform a deep clone.
- Around line 160-166: The async handlers handleRequestFinished and
handleRequestFailed can produce unhandled rejections when passed directly to
page.on; change them to ensure all exceptions are caught by wrapping the async
work in a try/catch (inside each function) or replace the direct registration
with a non-async wrapper that calls the async function and calls
.catch(...)—e.g., catch and log errors (or forward to commitRequest error
handling) around the call to commitRequest(request, await request.response()) so
failures are handled instead of propagating unhandled rejections; reference
handleRequestFinished, handleRequestFailed, and commitRequest when making the
change.

In `@packages/benchmark/src/perfume-collector.ts`:
- Line 168: Replace the fixed sleep call
page.waitForTimeout(BENCHMARK_CONSTANTS.PERFUME_METRICS_WAIT) with a
deterministic wait that polls for the actual metrics to be present; for example,
use page.waitForFunction to wait until the Perfume metrics object/DOM is
populated (inspect the collector code to see whether it exposes
window.__perfume, window.perfume, or a specific DOM element) so the collector in
perfume-collector.ts continues only when the actual metrics exist rather than
after a fixed timeout.
- Around line 242-253: The object built from rawMetrics uses inconsistent falsy
handling: replace the uses of || 0 with nullish coalescing so legitimate zero
values are preserved. Specifically, update the fields that reference rawMetrics
(firstPaint, firstContentfulPaint, largestContentfulPaint,
cumulativeLayoutShift, totalBlockingTime) to use rawMetrics.<metric>?.value ??
0, and change timeToFirstByte to use rawMetrics.TTFB?.value ??
navigationTiming?.timeToFirstByte ?? 0 so navigationTiming and
defaultNavigationTiming behavior remains intact; keep firstInputDelay and
interactionToNextPaint as ?? null.

In `@packages/benchmark/src/resource-timing-collector.ts`:
- Around line 192-197: The stylesheet detection filters (where styleEntries is
created from resourceEntries) currently uses entry.name.endsWith(".css") which
misses URLs with query strings or fragments (e.g., styles.css?v=1); update the
predicate to parse the entry.name as a URL (or strip the query/fragment) and
check the pathname for a trailing ".css" (or use a regex that tests for
\.css($|[?#])) so CSS files with query strings are correctly classified; modify
the filter creating styleEntries accordingly.

In `@packages/cookiebench-cli/README.md`:
- Line 377: Change the phrase "Open source status" to use a hyphenated compound
adjective so it reads "Open-source status" in the "4. **Transparency** (15%):
Open source status, documentation, licensing" line (update the specific text
"Open source status" to "Open-source status").

In `@packages/cookiebench-cli/src/commands/benchmark.ts`:
- Around line 80-84: The two optional timing fields are inconsistent:
timeToFirstByte returns undefined when no valid values but
interactionToNextPaint returns null; update the interactionToNextPaint
assignment to use undefined instead of null so both fields use the same
missing-value sentinel (e.g., change the interactionToNextPaint ternary to
return undefined when validInp.length === 0); ensure you keep
calculateAverage(validInp) for the present-case and reference the
timeToFirstByte and interactionToNextPaint properties and calculateAverage
helper when making the change.
- Around line 340-362: The current code mutates the loaded config object
in-place when applying iterations and profile/cacheMode overrides (the block
adjusting config.iterations and config.measurement.minSuccessfulIterations and
the subsequent profile/cacheMode overrides); to avoid side effects, create a
cloned config (e.g., clonedConfig = structuredClone(config) or a deep clone)
immediately after obtaining the config from loadValidatedConfigSync and apply
all mutations to clonedConfig instead of config, then pass clonedConfig into the
rest of the benchmark flow and update the debug log to reference clonedConfig;
ensure you clone deeply so nested fields like
measurement.minSuccessfulIterations are isolated.
- Around line 392-403: The current mkdir(tracesDir, { recursive: true }) catch
block can silently swallow errors that lack a code property; update the handler
in benchmark.ts so that any error without code or with a code different from
"EEXIST" is rethrown. In other words, extract the error code safely (e.g., const
code = (error as any)?.code) and only ignore the error when code === "EEXIST";
for all other cases (including undefined code) rethrow the error so unexpected
failures from mkdir are not swallowed.

In `@packages/cookiebench-cli/src/commands/db.ts`:
- Around line 156-166: The function generateCommand is declared synchronously
while similar handlers (pushCommand, migrateCommand, statusCommand) are async;
change generateCommand to an async function (async function
generateCommand(logger: CliLogger, projectRoot: string)) returning a
Promise<void> so its signature matches the others, keep the same internal logic
(calls to runDrizzleCommand, logger.*) and verify callers that await
generateCommand remain correct; this keeps consistency with
pushCommand/migrateCommand/statusCommand and avoids confusion about awaiting
non-Promise values.
- Around line 182-193: runDrizzleCommand currently calls process.exit(1) on
failure, so the try-catch in db.ts around runDrizzleCommand is unreachable;
refactor by removing process.exit from runDrizzleCommand and instead throw an
Error (or reject the promise) when it fails, then keep the existing try {
runDrizzleCommand(...) } catch (error) in db.ts to log via logger.error and call
process.exit(1) there; ensure runDrizzleCommand's signature returns/throws an
Error so callers can handle failures and do not swallow exits inside the helper.
- Around line 195-208: The studioCommand function should be made async (matching
generateCommand) and should await runDrizzleCommand; replace the empty catch
with error discrimination: in the catch inspect the thrown error (e.g.,
err.signal === 'SIGINT' or err.name === 'AbortError') and treat those as normal
interruptions by logging "Studio closed.", otherwise call logger.error with a
descriptive message and the error (or rethrow) so unexpected failures aren't
swallowed; reference studioCommand and runDrizzleCommand when making these
changes.
- Around line 61-62: The artificial 1-second pause after clearing the logger
(the await setTimeout(ONE_SECOND) following logger.clear()) adds unnecessary
latency; remove that await call (or replace ONE_SECOND with a much shorter
constant or make the delay configurable) so the screen clears immediately.
Update any surrounding logic that assumed the delay and run tests/CLI flows to
confirm no race conditions remain.
- Around line 255-256: The dynamic import of node:fs/promises for readdir is
unnecessary; replace it with a static top-level import and remove the
in-function dynamic import. Add "import { readdir } from 'node:fs/promises';" to
the file's existing imports and delete the line "const { readdir } = await
import('node:fs/promises');" so the code uses the top-level readdir symbol
directly where it's listed/used in the db.ts command logic.
- Around line 210-226: The findProjectRoot() function is duplicated; extract it
into a shared utility (e.g., a new projectRoot() exported from `@consentio/shared`
or a CLI utils module) and replace the local implementation in
packages/cookiebench-cli/src/commands/db.ts with an import of that shared
function; update any callers to use the new exported symbol (projectRoot or
findProjectRoot) and remove the local function to avoid duplication and keep
behavior consistent across packages.

In `@packages/cookiebench-cli/src/commands/migrate-results.ts`:
- Around line 125-129: The KB-detection heuristic (variables likelyKbData,
totalSizeCandidates, multiplier) can misclassify edge cases; update it to first
look for an explicit v1 marker (e.g., a stored sizeUnit or version flag on the
record) and use that when present, and if no marker exists fall back to a safer
heuristic: require both a majority of values < 100_000 AND a small max/min
spread (or a threshold like max < 200_000 and mean < 50_000) before setting
multiplier = 1024; whenever the fallback heuristic is used, emit a clear
logger.warn that the heuristic was applied and include summary stats (count,
min, max, mean) so callers can opt to re-run with an explicit flag if needed.
- Around line 262-263: The migration currently overwrites files in place (see
migrateV1ToV2, parsed, migrated, writeFile, file) so create a backup before
writing: copy or rename the original (file) to a safe backup name (e.g., file +
'.bak' or include timestamp) and only proceed to writeStringified migrated data
after the backup succeeds; on write failure, restore the backup and surface an
error, and consider writing to a temp file and renaming atomically to avoid
partial files. Ensure the backup/restore logic is added around the existing
writeFile call and that errors during backup or write are handled and reported.

In `@packages/cookiebench-cli/src/commands/results.ts`:
- Around line 366-876: The printDetailedResults function currently assumes
results has at least one iteration, causing divide-by-zero/NaN when
results.length === 0; add an early guard at the top of printDetailedResults that
checks if !results || results.length === 0, logs a brief message (or color.dim
warning) and returns early to avoid computing averages and rendering tables
(affects avg* calculations, totalFiles, waterfall/sampleResult logic and
aggregatedResources). Ensure the guard prevents further use of sampleResult,
firstResult and any reductions that divide by results.length.

In `@packages/cookiebench-cli/src/commands/save.ts`:
- Around line 101-139: The saveBenchmarkResult function currently calls fetch
without a timeout; wrap the request in an AbortController with a 15s timer:
create an AbortController, pass controller.signal to fetch, and set a setTimeout
that calls controller.abort() after 15000ms; clear that timeout after fetch
resolves (both success and failure) to avoid leaks. Update the catch in
saveBenchmarkResult to detect an abort (error.name === "AbortError" or
error.message indicating abort) and log a specific timeout/connection message
(using apiUrl), then rethrow; reference saveBenchmarkResult, logger, and
response handling when making these changes.

In `@packages/cookiebench-cli/src/commands/scores.ts`:
- Around line 330-337: The average calculation for cookieBannerVisibleTimeMs can
produce NaN when both b.timing.cookieBanner.userVisibleTime and
b.timing.cookieBanner.visibilityTime are null/undefined; update the reducer in
scores.ts (the cookieBannerVisibleTimeMs computation using appResults.reduce) to
coalesce missing timings to a numeric fallback (e.g., 0) or skip those entries
and adjust the divisor accordingly so the sum never becomes NaN; ensure you
reference cookieBannerVisibleTimeMs and the appResults.reduce call when making
the change.

In `@packages/cookiebench-cli/src/components/intro.ts`:
- Around line 42-82: The gradient currently emits raw ANSI escapes via
customColor and thus ignores NO_COLOR/TTY; update the logic to gate ANSI output
on picocolors' color support (e.g., check color.isColorSupported or
process.env.NO_COLOR/tty.isatty) so when color is not supported you return plain
figletText lines unchanged. Concretely, modify the use of customColor (or
initialize customColor) in the intro rendering path that handles figletText,
lines, gradientDenominator, and coloredLines so that if colors are unsupported
the mapping returns the original line (no ANSI) or make each customColor.* a
no-op when color support is false.

In `@packages/cookiebench-cli/src/types/index.ts`:
- Line 12: The grade/type inconsistency: the top-level type "grade" currently
uses PascalCase string literals ("Excellent" | "Good" | "Fair" | "Poor" |
"Critical") while "status" uses lowercase and omits "critical", which will break
mappings; fix this by extracting a single shared union type (e.g., GradeValue or
Grade) and use it for both the top-level grade and status representations (or
create a canonical lowercased union and consistently map/transform where
needed), update the "status" union to include "critical" if you keep lowercase,
or convert "grade" to lowercase to match, and adjust any code that reads/writes
these fields to use the shared type (refer to the grade and status type
declarations to locate where to change).

In `@packages/cookiebench-cli/src/utils/config-validation.ts`:
- Around line 424-445: The array branch treats "iffe" as invalid because
assertStringArray checks items against BUNDLE_TYPES while the string branch
first normalizes "iffe" to "iife"; update the array handling to normalize items
before validation (or accept "iffe" in BUNDLE_TYPES). Specifically, in the
techStack.bundleType Array path use the same normalization logic as the string
path (or call normalizeBundleType on each array element) prior to calling
assertStringArray so that values like "iffe" are normalized to "iife" and
validated against BUNDLE_TYPES consistently.
- Around line 286-296: Simplify the complex conditional that assigns enabled by
first calling assertBoolean and capturing its result, then use the captured
boolean to decide behavior: call assertBoolean(issues, "remote.enabled",
remote.enabled) only if remote.enabled is not undefined and store the result in
a variable (e.g., isValidEnabled), then set enabled = remote.enabled &&
isValidEnabled (or false if remote.enabled is undefined). After that, keep the
existing checks for assertString(issues, "remote.url", remote.url) when enabled
is true or when remote.url is provided. Reference symbols: enabled,
remote.enabled, assertBoolean, isValidEnabled (new temp), assertString,
remote.url, issues.

In `@packages/cookiebench-cli/src/utils/project-root.ts`:
- Around line 32-37: The function currently returns join(projectRoot,
"benchmarks", appPath) without verifying it contains config.json; update the
logic (using directPath, projectRoot, appPath, existsSync, join) to check the
fallback path for a config.json as well and only return it if
existsSync(join(fallbackPath, "config.json")) is true, otherwise throw or return
a clear error/undefined so callers don't get an invalid benchmark path.

In `@packages/cookiebench-cli/src/utils/scoring.ts`:
- Around line 507-514: The calculateNetworkScore function currently declares an
unused parameter named _resourceData; either remove that parameter from the
function signature (and all callers) to eliminate dead code, or if you intend to
keep it for future use, leave the parameter but add a brief comment next to
_resourceData explaining it is reserved for future use and should remain unused
(e.g., "reserved for future resource-based scoring"); update the function
signature and any references accordingly to keep linting/tests clean.
- Around line 186-207: The parseTechStack function returns techStack.typescript
directly which can be undefined; change it to default to false so parsed results
are consistent with the catch fallback. Update the return in parseTechStack (the
success path) to use a boolean coalescing/default (e.g., set typescript to
techStack.typescript ?? false) so that typescript is always false when missing
or undefined.
- Around line 153-163: The open-source tag check in the block that computes
lowerTags (variable lowerTags and the conditional returning true) is too broad
because it treats "free" and "community" as definitive indicators; remove "free"
and "community" from the includes list (or replace them with more specific
phrases like "community edition" or "free and open source" if you want to retain
some coverage) so the conditional only matches clear OSS indicators ("open
source", "opensource", "oss"); update the conditional in the same function (the
lowerTags.includes(...) checks) accordingly.
- Around line 1354-1361: The category and detail score displays hardcode “/100”
which is wrong for items with varying maxScore; update the table rows in the
scores rendering logic so category rows use
`${category.score}/${category.maxScore}` (replace `${category.score}/100`) and
detail rows use `${detail.score}/${detail.maxScore}` (replace
`${detail.score}/100`) so each display shows the actual maxScore for that item;
locate these changes around the loop over scores.categories and the inner loop
that pushes detail rows (references: scores.categories, category.score,
category.maxScore, detail.score, detail.maxScore).

In `@packages/cookiebench-cli/tsconfig.base.json`:
- Line 9: Remove the browser DOM libs from the TypeScript lib list: edit the
tsconfig's "lib" array (the "lib" property containing "es2022", "DOM",
"DOM.Iterable") and delete the "DOM" and "DOM.Iterable" entries so the CLI
package uses only Node-oriented libs (leaving "es2022" or other Node-targeted
entries intact).

In `@packages/runner/package.json`:
- Around line 1-33: Add a top-level "engines" field to this package.json for
`@consentio/runner` to match the monorepo constraint; specifically add "engines":
{ "node": ">=20.9.0" } at the root of the manifest (alongside "name", "version",
"type", etc.) so the runner package enforces the same Node version requirement
as the benchmarks package.

In `@packages/runner/README.md`:
- Around line 44-62: The README example and API list call buildAndServeNextApp
without the required logger parameter; update the usage example and the API
documentation to pass a logger instance (e.g., processLogger or any object
implementing the expected logger interface) to buildAndServeNextApp(serverPath,
logger) so the call in the snippet and the API signature reflect the new
required logger argument; reference buildAndServeNextApp in the README example
and the API section so users know to provide the logger.

In `@packages/runner/src/benchmark-runner.ts`:
- Around line 446-448: The call to global.gc in benchmark-runner.ts can be a
no-op unless Node was started with --expose-gc; update the code around the
global.gc() call to detect and handle that case: check for global.gc and
process.execArgv.includes('--expose-gc') (or at least global.gc existence) and
when absent emit a clear warning via the existing logger (or console.warn)
indicating GC is unavailable and that Node must be started with --expose-gc to
enable manual GC; keep the current global.gc() invocation when available but add
the log/documentation path when it is not.
- Around line 312-324: The execFileAsync call using traceZipPath and tempDir (in
benchmark-runner.ts around execFileAsync, traceDir, traceZipPath, traceFilePath)
can be fed untrusted path components; validate and sanitize those paths before
invoking execFileAsync: resolve and normalize traceZipPath and tempDir (e.g.,
via path.resolve/path.normalize), reject null-bytes or path-traversal (../) and
ensure the resolved paths are inside an allowed base directory (or a created
safe temp directory) and exist as expected; restrict config-derived names to a
safe character whitelist or generate safe filenames yourself and use those, then
call execFileAsync with the sanitized absolute paths and only the expected
filename ("trace.trace") to prevent injection or unexpected file access.
- Around line 66-70: The network profiles `fast3g` and `slow4g` in
benchmark-runner.ts have mismatched throughput/latency values; update them to
realistic values (or add a clarifying comment if the divergence is intentional).
Specifically, change `slow4g` to a typical 4G profile (e.g., latency ~60ms,
download ~10 Mbps, upload ~3 Mbps) and change `fast3g` to a typical 3G profile
(e.g., latency ~120ms, download ~1.6 Mbps, upload ~0.75 Mbps); locate the
`fast3g` and `slow4g` objects and replace the numeric literals for `latency`,
`downloadThroughput`, and `uploadThroughput` accordingly (or add an explanatory
comment above each profile if you want to keep synthetic/conservative values).

In `@packages/runner/src/performance-aggregator.ts`:
- Around line 276-300: The comment notes a mismatch between the AI summary and
the code: the calculateNetworkImpact method is declared private but referenced
as public; update the method visibility or docs accordingly. Change the method
declaration name calculateNetworkImpact in class PerformanceAggregator (or the
file performance-aggregator.ts) from private to public if it must be used by
external callers like BenchmarkRunner, or if it should remain internal, update
the AI summary/docs to state it is private/internal; ensure any callers (e.g.,
BenchmarkRunner) are updated to use the public API or tests adjusted to reflect
the correct visibility.
- Around line 302-314: The method hasMeaningfulVariability is declared private
but the AI summary and external callers expect it to be public; change its
visibility from private to public (i.e., make the method public on the class
where calculateStatistics(...) and hasMeaningfulVariability are declared),
update the method signature accordingly, and run/adjust any unit tests or
imports that reference it so external consumers can call
hasMeaningfulVariability(...) directly; ensure exported class typings reflect
the public method.

In `@packages/runner/src/server.ts`:
- Around line 48-69: The child process is created with stdout piped but not
consumed, which can block; update the spawn call that creates serverProcess
(spawn(pm.command, startArgs, {...})) to either inherit stdout by changing stdio
from ["inherit","pipe","inherit"] to ["inherit","inherit","inherit"] (or
similar) or explicitly consume serverProcess.stdout by attaching a
data/error/close listener that forwards/drains output (e.g., piping to
process.stdout or reading and discarding data) so the buffer cannot fill during
Next.js startup.

In `@packages/runner/src/statistics.ts`:
- Around line 150-152: The current check if (mean === 0) { return 0; } in the CV
calculation masks an edge case; update the CV function in statistics.ts (the
routine that computes coefficient of variation using the mean variable and CV
result) to return NaN instead of 0 when mean === 0, add a short comment
explaining this choice (signals undefined CV for zero mean), and ensure any
callers of this function handle NaN (or explicitly document the behavior) so
consumers can detect and handle the edge case.
- Around line 45-47: The variance is computed as a population variance (dividing
by values.length) which underestimates variability for small samples; change the
computation in statistics.ts where variance and stddev are calculated (variables
variance, stddev using values and mean) to use sample variance by dividing by
(values.length - 1) when values.length > 1 (and handle length <=1 safely), and
mirror the same change at the other occurrence around lines 146-147 to keep
consistency across the file.

In `@packages/shared/package.json`:
- Line 25: The `@types/node` dependency in package.json is set to ^24 which may
not match the project's supported Node range (benchmarks/onetrust specifies node
>=20.9.0); update the `@types/node` major to align with your minimum Node major
(e.g., ^20) or use the project's dependency catalog reference consistent with
other packages, and ensure package.json's "dependencies"/"devDependencies" entry
for "@types/node" is changed (search for the string "@types/node" in the file)
so the type definitions don't expose APIs newer than your supported Node
runtime.

In `@packages/shared/src/constants.ts`:
- Around line 5-7: BYTES_TO_KB and KILOBYTE are duplicate constants (both 1024);
remove the redundancy by keeping one canonical identifier and updating all
usages accordingly — choose either BYTES_TO_KB (if you prefer explicit
conversion naming) or KILOBYTE (if you prefer unit naming), then refactor
references to the removed constant across the codebase (search for BYTES_TO_KB
and KILOBYTE) and update imports/exports to use the retained constant
consistently; if both concepts are required, rename one to clarify semantic
difference (e.g., BYTES_PER_KILOBYTE) and adjust usages to match.
- Around line 9-11: PERCENTAGE_MULTIPLIER and PERCENTAGE_DIVISOR are both set to
100, which is redundant; either consolidate them into a single constant (e.g.,
PERCENTAGE_SCALE or PERCENTAGE_FACTOR) and update all usages of
PERCENTAGE_MULTIPLIER and PERCENTAGE_DIVISOR to that single symbol, or keep both
but add clear comments explaining the distinct semantic roles (multiplying to
convert to percent vs dividing to convert to decimal) and ensure names reflect
those roles; update any imports/usages of PERCENTAGE_MULTIPLIER and
PERCENTAGE_DIVISOR to the chosen symbol and run tests to confirm no regressions.
- Around line 13-14: TTI_BUFFER_MS is hardcoded to 1000 while ONE_SECOND already
exists; update the constant so TTI_BUFFER_MS references ONE_SECOND instead of
the literal to make the relationship explicit and prevent divergence (locate the
constants ONE_SECOND and TTI_BUFFER_MS in constants.ts and replace the numeric
literal with ONE_SECOND).

In `@packages/shared/src/utils/config.ts`:
- Around line 27-31: The catch block in the function that reads config (the
catch handling the read of configPath / join(process.cwd(), "config.json")) uses
console.error which will conflict with the ultracite “No console” rule; replace
the console.error call with the shared logging utility (e.g., logger.error or
getLogger(...).error) or add a rule-allowlist comment if intentional; update the
error logging call in that catch to forward the same message and error object
via the shared logger (preserving the `${configPath || join(process.cwd(),
"config.json")}` context) so the lint rule is satisfied and debugging info is
preserved.
📜 Review details

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between fa06799 and 38a3c34.

⛔ Files ignored due to path filters (12)
  • benchmarks/c15t-nextjs/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/c15t-react/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/cookie-control/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/cookie-yes/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/didomi/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/enzuzo/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/iubenda/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/ketch/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/onetrust/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/osano/app/favicon.ico is excluded by !**/*.ico
  • benchmarks/usercentrics/app/favicon.ico is excluded by !**/*.ico
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (192)
  • .cursor/rules/ultracite.mdc
  • .gitignore
  • METHODOLOGY.md
  • README.md
  • benchmarks/baseline/config.json
  • benchmarks/baseline/next-env.d.ts
  • benchmarks/baseline/package.json
  • benchmarks/c15t-nextjs/app/layout.tsx
  • benchmarks/c15t-nextjs/app/page.tsx
  • benchmarks/c15t-nextjs/config.json
  • benchmarks/c15t-nextjs/next-env.d.ts
  • benchmarks/c15t-nextjs/next.config.ts
  • benchmarks/c15t-nextjs/package.json
  • benchmarks/c15t-nextjs/tsconfig.json
  • benchmarks/c15t-react/app/layout.tsx
  • benchmarks/c15t-react/app/page.tsx
  • benchmarks/c15t-react/config.json
  • benchmarks/c15t-react/next-env.d.ts
  • benchmarks/c15t-react/next.config.ts
  • benchmarks/c15t-react/package.json
  • benchmarks/c15t-react/tsconfig.json
  • benchmarks/cookie-control/app/layout.tsx
  • benchmarks/cookie-control/app/page.tsx
  • benchmarks/cookie-control/config.json
  • benchmarks/cookie-control/next-env.d.ts
  • benchmarks/cookie-control/next.config.ts
  • benchmarks/cookie-control/package.json
  • benchmarks/cookie-control/tsconfig.json
  • benchmarks/cookie-yes/app/layout.tsx
  • benchmarks/cookie-yes/app/page.tsx
  • benchmarks/cookie-yes/config.json
  • benchmarks/cookie-yes/next-env.d.ts
  • benchmarks/cookie-yes/next.config.ts
  • benchmarks/cookie-yes/package.json
  • benchmarks/cookie-yes/tsconfig.json
  • benchmarks/didomi/app/layout.tsx
  • benchmarks/didomi/app/page.tsx
  • benchmarks/didomi/config.json
  • benchmarks/didomi/next-env.d.ts
  • benchmarks/didomi/next.config.ts
  • benchmarks/didomi/package.json
  • benchmarks/didomi/tsconfig.json
  • benchmarks/enzuzo/app/layout.tsx
  • benchmarks/enzuzo/app/page.tsx
  • benchmarks/enzuzo/config.json
  • benchmarks/enzuzo/next-env.d.ts
  • benchmarks/enzuzo/next.config.ts
  • benchmarks/enzuzo/package.json
  • benchmarks/enzuzo/tsconfig.json
  • benchmarks/iubenda/app/layout.tsx
  • benchmarks/iubenda/app/page.tsx
  • benchmarks/iubenda/config.json
  • benchmarks/iubenda/next-env.d.ts
  • benchmarks/iubenda/next.config.ts
  • benchmarks/iubenda/package.json
  • benchmarks/iubenda/tsconfig.json
  • benchmarks/ketch/app/layout.tsx
  • benchmarks/ketch/app/page.tsx
  • benchmarks/ketch/config.json
  • benchmarks/ketch/next-env.d.ts
  • benchmarks/ketch/next.config.ts
  • benchmarks/ketch/package.json
  • benchmarks/ketch/tsconfig.json
  • benchmarks/onetrust/app/layout.tsx
  • benchmarks/onetrust/app/page.tsx
  • benchmarks/onetrust/config.json
  • benchmarks/onetrust/next-env.d.ts
  • benchmarks/onetrust/next.config.ts
  • benchmarks/onetrust/package.json
  • benchmarks/onetrust/tsconfig.json
  • benchmarks/osano/app/layout.tsx
  • benchmarks/osano/app/page.tsx
  • benchmarks/osano/config.json
  • benchmarks/osano/next-env.d.ts
  • benchmarks/osano/next.config.ts
  • benchmarks/osano/package.json
  • benchmarks/osano/tsconfig.json
  • benchmarks/usercentrics/app/layout.tsx
  • benchmarks/usercentrics/app/page.tsx
  • benchmarks/usercentrics/config.json
  • benchmarks/usercentrics/next-env.d.ts
  • benchmarks/usercentrics/next.config.ts
  • benchmarks/usercentrics/package.json
  • benchmarks/usercentrics/tsconfig.json
  • benchmarks/with-c15t-nextjs/package.json
  • benchmarks/with-c15t-react/package.json
  • benchmarks/with-cookie-control/config.json
  • benchmarks/with-cookie-control/next.config.ts
  • benchmarks/with-cookie-control/tsconfig.json
  • benchmarks/with-didomi/app/layout.tsx
  • benchmarks/with-didomi/next-env.d.ts
  • benchmarks/with-enzuzo/app/layout.tsx
  • benchmarks/with-enzuzo/config.json
  • benchmarks/with-enzuzo/next-env.d.ts
  • benchmarks/with-enzuzo/tsconfig.json
  • benchmarks/with-iubenda/next-env.d.ts
  • benchmarks/with-ketch/next-env.d.ts
  • benchmarks/with-onetrust/next-env.d.ts
  • benchmarks/with-onetrust/package.json
  • benchmarks/with-osano/next-env.d.ts
  • benchmarks/with-usercentrics/next-env.d.ts
  • package.json
  • packages/benchmark-schema/package.json
  • packages/benchmark-schema/results.schema.json
  • packages/benchmark-schema/schema.json
  • packages/benchmark/README.md
  • packages/benchmark/package.json
  • packages/benchmark/rslib.config.ts
  • packages/benchmark/src/bundle-strategy.ts
  • packages/benchmark/src/constants.ts
  • packages/benchmark/src/cookie-banner-collector.ts
  • packages/benchmark/src/index.ts
  • packages/benchmark/src/network-monitor.ts
  • packages/benchmark/src/perfume-collector.ts
  • packages/benchmark/src/resource-timing-collector.ts
  • packages/benchmark/src/types.ts
  • packages/benchmark/tsconfig.json
  • packages/cli/base.json
  • packages/cli/package.json
  • packages/cli/rslib.config.ts
  • packages/cli/src/commands/benchmark/benchmark-runner.ts
  • packages/cli/src/commands/benchmark/bundle-strategy.ts
  • packages/cli/src/commands/benchmark/constants.ts
  • packages/cli/src/commands/benchmark/cookie-banner-detector.ts
  • packages/cli/src/commands/benchmark/index.ts
  • packages/cli/src/commands/benchmark/metrics-calculator.ts
  • packages/cli/src/commands/benchmark/network-monitor.ts
  • packages/cli/src/commands/benchmark/resource-collector.ts
  • packages/cli/src/commands/benchmark/types.ts
  • packages/cli/src/commands/db.ts
  • packages/cli/src/commands/results.ts
  • packages/cli/src/index.ts
  • packages/cli/src/lib/README.md
  • packages/cli/src/lib/benchmark-runner.ts
  • packages/cli/src/lib/collectors/cookie-banner-collector.ts
  • packages/cli/src/lib/collectors/index.ts
  • packages/cli/src/lib/collectors/network-monitor.ts
  • packages/cli/src/lib/collectors/resource-timing-collector.ts
  • packages/cli/src/lib/metrics/index.ts
  • packages/cli/src/lib/metrics/performance-aggregator.ts
  • packages/cli/src/lib/performance-enhanced.ts
  • packages/cli/src/lib/performance.ts
  • packages/cli/src/lib/server.ts
  • packages/cli/src/types/index.ts
  • packages/cli/src/utils/index.ts
  • packages/cli/src/utils/scoring.ts
  • packages/cli/tsconfig.json
  • packages/cookiebench-cli/README.md
  • packages/cookiebench-cli/package.json
  • packages/cookiebench-cli/rslib.config.ts
  • packages/cookiebench-cli/src/commands/benchmark.ts
  • packages/cookiebench-cli/src/commands/db.ts
  • packages/cookiebench-cli/src/commands/migrate-results.ts
  • packages/cookiebench-cli/src/commands/results.ts
  • packages/cookiebench-cli/src/commands/save.ts
  • packages/cookiebench-cli/src/commands/scores.ts
  • packages/cookiebench-cli/src/components/intro.ts
  • packages/cookiebench-cli/src/index.ts
  • packages/cookiebench-cli/src/types/index.ts
  • packages/cookiebench-cli/src/utils/auth.ts
  • packages/cookiebench-cli/src/utils/config-validation.ts
  • packages/cookiebench-cli/src/utils/constants.ts
  • packages/cookiebench-cli/src/utils/index.ts
  • packages/cookiebench-cli/src/utils/logger.ts
  • packages/cookiebench-cli/src/utils/project-root.ts
  • packages/cookiebench-cli/src/utils/scoring.ts
  • packages/cookiebench-cli/tsconfig.base.json
  • packages/cookiebench-cli/tsconfig.json
  • packages/runner/README.md
  • packages/runner/package.json
  • packages/runner/rslib.config.ts
  • packages/runner/src/benchmark-runner.ts
  • packages/runner/src/index.ts
  • packages/runner/src/performance-aggregator.ts
  • packages/runner/src/server.ts
  • packages/runner/src/statistics.ts
  • packages/runner/src/types.ts
  • packages/runner/src/utils.ts
  • packages/runner/tsconfig.json
  • packages/shared/README.md
  • packages/shared/package.json
  • packages/shared/rslib.config.ts
  • packages/shared/src/constants.ts
  • packages/shared/src/index.ts
  • packages/shared/src/utils/config.ts
  • packages/shared/src/utils/conversion.ts
  • packages/shared/src/utils/package-manager.ts
  • packages/shared/src/utils/time.ts
  • packages/shared/tsconfig.json
  • pnpm-workspace.yaml
  • tsconfig.json
  • turbo.json
💤 Files with no reviewable changes (47)
  • benchmarks/with-c15t-nextjs/package.json
  • packages/cli/src/commands/benchmark/constants.ts
  • packages/cli/src/lib/collectors/index.ts
  • packages/cli/src/index.ts
  • benchmarks/with-iubenda/next-env.d.ts
  • benchmarks/with-didomi/app/layout.tsx
  • packages/cli/src/commands/benchmark/benchmark-runner.ts
  • benchmarks/with-enzuzo/app/layout.tsx
  • benchmarks/with-ketch/next-env.d.ts
  • packages/cli/src/lib/metrics/performance-aggregator.ts
  • benchmarks/with-usercentrics/next-env.d.ts
  • packages/cli/src/lib/metrics/index.ts
  • packages/cli/src/types/index.ts
  • packages/cli/src/lib/collectors/cookie-banner-collector.ts
  • packages/cli/src/commands/benchmark/resource-collector.ts
  • packages/cli/rslib.config.ts
  • packages/cli/src/commands/benchmark/index.ts
  • benchmarks/with-enzuzo/tsconfig.json
  • benchmarks/with-didomi/next-env.d.ts
  • packages/cli/src/commands/benchmark/metrics-calculator.ts
  • packages/cli/src/lib/collectors/network-monitor.ts
  • packages/cli/src/commands/results.ts
  • packages/cli/src/commands/benchmark/types.ts
  • packages/cli/src/lib/server.ts
  • packages/cli/tsconfig.json
  • packages/cli/src/lib/README.md
  • benchmarks/with-enzuzo/config.json
  • benchmarks/with-onetrust/package.json
  • packages/cli/src/commands/benchmark/network-monitor.ts
  • packages/cli/src/lib/performance-enhanced.ts
  • benchmarks/with-cookie-control/config.json
  • packages/cli/src/lib/benchmark-runner.ts
  • packages/cli/src/commands/benchmark/cookie-banner-detector.ts
  • benchmarks/with-enzuzo/next-env.d.ts
  • packages/cli/src/utils/scoring.ts
  • benchmarks/with-cookie-control/tsconfig.json
  • packages/cli/src/lib/collectors/resource-timing-collector.ts
  • benchmarks/with-cookie-control/next.config.ts
  • packages/cli/src/commands/db.ts
  • benchmarks/with-osano/next-env.d.ts
  • packages/cli/package.json
  • packages/cli/src/utils/index.ts
  • packages/cli/src/commands/benchmark/bundle-strategy.ts
  • packages/cli/src/lib/performance.ts
  • benchmarks/with-onetrust/next-env.d.ts
  • benchmarks/with-c15t-react/package.json
  • packages/cli/base.json
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-11-07T16:17:07.589Z
Learnt from: danielsitek
Repo: consentdotio/benchmarks PR: 6
File: benchmarks/with-dgp-cookie-consent/config.json:15-20
Timestamp: 2025-11-07T16:17:07.589Z
Learning: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").

Applied to files:

  • packages/cookiebench-cli/package.json
  • benchmarks/ketch/config.json
  • packages/benchmark/README.md
  • benchmarks/cookie-control/config.json
  • benchmarks/cookie-control/package.json
  • benchmarks/enzuzo/config.json
  • packages/benchmark/src/bundle-strategy.ts
  • package.json
  • benchmarks/usercentrics/config.json
  • benchmarks/onetrust/config.json
  • packages/cookiebench-cli/src/types/index.ts
  • benchmarks/cookie-yes/config.json
  • benchmarks/c15t-nextjs/config.json
  • packages/benchmark-schema/results.schema.json
  • packages/benchmark/package.json
  • benchmarks/cookie-yes/package.json
  • benchmarks/baseline/package.json
  • packages/cookiebench-cli/rslib.config.ts
  • packages/benchmark/src/constants.ts
  • README.md
  • packages/runner/src/index.ts
  • packages/benchmark/src/index.ts
  • benchmarks/didomi/config.json
  • benchmarks/c15t-nextjs/package.json
  • packages/benchmark/tsconfig.json
  • packages/cookiebench-cli/tsconfig.base.json
  • packages/benchmark/src/types.ts
  • packages/runner/src/types.ts
  • benchmarks/c15t-react/config.json
  • benchmarks/osano/config.json
  • packages/cookiebench-cli/tsconfig.json
  • benchmarks/iubenda/config.json
  • packages/benchmark-schema/package.json
  • benchmarks/iubenda/package.json
  • packages/benchmark-schema/schema.json
📚 Learning: 2025-11-07T16:19:37.939Z
Learnt from: danielsitek
Repo: consentdotio/benchmarks PR: 6
File: benchmarks/with-dgp-cookie-consent/config.json:37-39
Timestamp: 2025-11-07T16:19:37.939Z
Learning: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the includes.backend property accepts either a string or an array of strings (oneOf), not a boolean.

Applied to files:

  • benchmarks/cookie-control/config.json
  • benchmarks/enzuzo/config.json
  • benchmarks/onetrust/config.json
  • benchmarks/cookie-yes/config.json
  • packages/benchmark-schema/results.schema.json
  • packages/benchmark-schema/package.json
  • packages/benchmark-schema/schema.json
🧬 Code graph analysis (11)
packages/cookiebench-cli/src/components/intro.ts (1)
packages/cookiebench-cli/src/utils/logger.ts (2)
  • logger (162-162)
  • CliLogger (9-9)
packages/runner/src/statistics.ts (1)
packages/runner/src/index.ts (4)
  • calculateStatistics (29-29)
  • calculateCoefficientOfVariation (28-28)
  • calculateTrimmedMean (30-30)
  • isStable (31-31)
packages/cookiebench-cli/src/utils/index.ts (4)
packages/runner/src/index.ts (2)
  • readConfig (34-34)
  • Config (16-16)
packages/runner/src/utils.ts (1)
  • readConfig (6-16)
packages/shared/src/utils/config.ts (1)
  • readConfig (20-35)
packages/benchmark/src/types.ts (1)
  • Config (30-75)
packages/shared/src/utils/config.ts (1)
packages/cookiebench-cli/src/utils/index.ts (1)
  • readConfig (28-30)
packages/runner/src/server.ts (3)
packages/runner/src/types.ts (1)
  • ServerInfo (22-25)
packages/runner/src/utils.ts (2)
  • getPackageManager (25-156)
  • ONE_SECOND (5-5)
packages/shared/src/utils/package-manager.ts (1)
  • getPackageManager (27-85)
packages/runner/src/utils.ts (1)
packages/benchmark/src/types.ts (1)
  • Config (30-75)
packages/cookiebench-cli/src/commands/results.ts (6)
packages/cookiebench-cli/src/types/index.ts (2)
  • BenchmarkScores (10-44)
  • Config (5-5)
packages/cookiebench-cli/src/utils/config-validation.ts (3)
  • loadValidatedConfigSync (564-592)
  • ConfigValidationError (36-44)
  • formatConfigIssues (560-562)
packages/runner/src/utils.ts (2)
  • formatTime (18-23)
  • ONE_SECOND (5-5)
packages/cookiebench-cli/src/utils/constants.ts (16)
  • SCORE_THRESHOLD_POOR (10-10)
  • SCORE_THRESHOLD_FAIR (11-11)
  • CLS_DECIMAL_PLACES (7-7)
  • CLS_THRESHOLD_GOOD (14-14)
  • CLS_THRESHOLD_FAIR (15-15)
  • COL_WIDTH_NAME (18-18)
  • COL_WIDTH_CHART_PADDING (24-24)
  • MAX_FILENAME_LENGTH (27-27)
  • TRUNCATED_FILENAME_LENGTH (28-28)
  • MIN_DURATION_THRESHOLD (31-31)
  • COL_WIDTH_TYPE (19-19)
  • COL_WIDTH_SOURCE (20-20)
  • COL_WIDTH_SIZE (21-21)
  • COL_WIDTH_DURATION (22-22)
  • COL_WIDTH_TAGS (23-23)
  • DEFAULT_DOM_SIZE (3-3)
packages/cookiebench-cli/src/utils/project-root.ts (1)
  • findProjectRoot (4-22)
packages/cookiebench-cli/src/utils/scoring.ts (1)
  • calculateScores (1044-1325)
packages/benchmark/src/cookie-banner-collector.ts (3)
packages/benchmark/src/types.ts (5)
  • Config (30-75)
  • CookieBannerMetrics (99-112)
  • WindowWithCookieMetrics (84-97)
  • LayoutShiftEntry (78-81)
  • CookieBannerData (114-129)
packages/benchmark/src/bundle-strategy.ts (1)
  • determineBundleStrategy (4-35)
packages/benchmark/src/constants.ts (1)
  • BENCHMARK_CONSTANTS (17-74)
packages/benchmark/src/constants.ts (3)
packages/benchmark/src/index.ts (1)
  • BENCHMARK_CONSTANTS (5-5)
packages/cookiebench-cli/src/utils/index.ts (1)
  • ONE_SECOND (13-13)
packages/runner/src/utils.ts (1)
  • ONE_SECOND (5-5)
packages/cookiebench-cli/src/commands/benchmark.ts (5)
packages/runner/src/types.ts (2)
  • BenchmarkResult (234-323)
  • ServerInfo (22-25)
packages/cookiebench-cli/src/utils/index.ts (2)
  • findProjectRoot (26-26)
  • resolveBenchmarkPath (26-26)
packages/cookiebench-cli/src/utils/constants.ts (3)
  • DEFAULT_DOM_SIZE (3-3)
  • DEFAULT_ITERATIONS (2-2)
  • SEPARATOR_WIDTH (5-5)
packages/runner/src/server.ts (1)
  • buildAndServeNextApp (6-122)
packages/cookiebench-cli/src/utils/project-root.ts (2)
  • findProjectRoot (4-22)
  • resolveBenchmarkPath (24-38)
benchmarks/osano/app/layout.tsx (2)
benchmarks/enzuzo/app/layout.tsx (2)
  • metadata (4-6)
  • RootLayout (8-25)
benchmarks/didomi/app/layout.tsx (1)
  • RootLayout (20-92)
🪛 ast-grep (0.40.5)
benchmarks/cookie-control/app/layout.tsx

[warning] 22-22: Usage of dangerouslySetInnerHTML detected. This bypasses React's built-in XSS protection. Always sanitize HTML content using libraries like DOMPurify before injecting it into the DOM to prevent XSS attacks.
Context: dangerouslySetInnerHTML
Note: [CWE-79] Improper Neutralization of Input During Web Page Generation [REFERENCES]
- https://reactjs.org/docs/dom-elements.html#dangerouslysetinnerhtml
- https://cwe.mitre.org/data/definitions/79.html

(react-unsafe-html-injection)

benchmarks/onetrust/app/layout.tsx

[warning] 23-23: Usage of dangerouslySetInnerHTML detected. This bypasses React's built-in XSS protection. Always sanitize HTML content using libraries like DOMPurify before injecting it into the DOM to prevent XSS attacks.
Context: dangerouslySetInnerHTML
Note: [CWE-79] Improper Neutralization of Input During Web Page Generation [REFERENCES]
- https://reactjs.org/docs/dom-elements.html#dangerouslysetinnerhtml
- https://cwe.mitre.org/data/definitions/79.html

(react-unsafe-html-injection)

benchmarks/iubenda/app/layout.tsx

[warning] 18-18: Usage of dangerouslySetInnerHTML detected. This bypasses React's built-in XSS protection. Always sanitize HTML content using libraries like DOMPurify before injecting it into the DOM to prevent XSS attacks.
Context: dangerouslySetInnerHTML
Note: [CWE-79] Improper Neutralization of Input During Web Page Generation [REFERENCES]
- https://reactjs.org/docs/dom-elements.html#dangerouslysetinnerhtml
- https://cwe.mitre.org/data/definitions/79.html

(react-unsafe-html-injection)

🪛 LanguageTool
METHODOLOGY.md

[grammar] ~79-~79: Ensure spelling is correct
Context: ...aintains speed - Fast 3G (1.6 Mbps, 562ms latency): More realistic, significantly...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

packages/cookiebench-cli/README.md

[uncategorized] ~377-~377: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...ice overhead 4. Transparency (15%): Open source status, documentation, licensing 5. **U...

(EN_COMPOUND_ADJECTIVE_INTERNAL)

@@ -1,5 +1,6 @@
/// <reference types="next" />
/// <reference types="next/image-types/global" />
import "./.next/types/routes.d.ts";
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if .next is gitignored in the benchmark directory
echo "=== Checking .gitignore in benchmarks/c15t-nextjs ==="
if [ -f benchmarks/c15t-nextjs/.gitignore ]; then
  cat benchmarks/c15t-nextjs/.gitignore | grep -E '\.next|^\.next$' || echo "Pattern '.next' not found in .gitignore"
else
  echo ".gitignore not found in benchmarks/c15t-nextjs"
fi

echo ""
echo "=== Checking root .gitignore ==="
if [ -f .gitignore ]; then
  cat .gitignore | grep -E '\.next|^\.next$' || echo "Pattern '.next' not found in root .gitignore"
else
  echo ".gitignore not found at root"
fi

echo ""
echo "=== Checking next-env.d.ts file ==="
cat -n benchmarks/c15t-nextjs/next-env.d.ts

echo ""
echo "=== Checking if .next directory exists ==="
ls -la benchmarks/c15t-nextjs/.next 2>/dev/null || echo ".next directory does not exist"

echo ""
echo "=== Checking if routes.d.ts exists ==="
fd -t f 'routes.d.ts' benchmarks/c15t-nextjs/ 2>/dev/null || echo "No routes.d.ts found"

Repository: consentdotio/benchmarks

Length of output: 649


🌐 Web query:

Next.js next-env.d.ts auto-generated routes.d.ts import app directory

💡 Result:

  • Don’t edit next-env.d.ts to “import” route types. next-env.d.ts is auto-regenerated by next dev, next build, or next typegen, so manual changes will get overwritten. Next.js expects it to be picked up via tsconfig.json include. [1]

  • If you mean Next.js “typed routes” (App Router):

    1. Enable it:
      // next.config.ts
      import type { NextConfig } from "next";
      const nextConfig: NextConfig = { typedRoutes: true };
      export default nextConfig;
      [1]
    2. Ensure TypeScript includes the generated type files (this is what makes the generated route types available—no import from app/ needed):
      // tsconfig.json
      {
        "include": [
          "next-env.d.ts",
          ".next/types/**/*.ts",
          "**/*.ts",
          "**/*.tsx"
        ]
      }
      [1]
    3. Use it via Next’s types (example):
      import type { Route } from "next";
      [1]
  • If you’re using a 3rd-party generator that creates routes.d.ts (or similar) (e.g. nextjs-routes):

    • These tools typically generate a .d.ts file (commonly under @types/…) and rely on TypeScript picking it up globally (again: via tsconfig / type resolution, not by importing it from app/). nextjs-routes specifically generates @types/nextjs-routes.d.ts. [2]

If you paste (1) where your routes.d.ts is being generated and (2) your tsconfig.json, I can tell you the exact include/typeRoots change needed.

Sources: [1] Next.js TypeScript docs (typed routes, .next/types, next-env.d.ts) https://nextjs.org/docs/app/api-reference/config/typescript, [2] nextjs-routes (generated @types/nextjs-routes.d.ts) https://github.com/tatethurston/nextjs-routes


🏁 Script executed:

#!/bin/bash
# Check if this is a Next.js generated file by looking at git history or similar markers
echo "=== Checking git status/history for next-env.d.ts ==="
cd benchmarks/c15t-nextjs
git log --oneline -n 5 next-env.d.ts 2>/dev/null || echo "No git history available"

echo ""
echo "=== Checking if .next/types directory would be created ==="
# Look for references in package.json or build scripts
if [ -f package.json ]; then
  echo "=== package.json content ==="
  cat package.json | grep -A 5 -B 5 "next" || echo "No next references"
fi

echo ""
echo "=== Checking Next.js version ==="
if [ -f package.json ]; then
  cat package.json | grep "next"
fi

Repository: consentdotio/benchmarks

Length of output: 1186


Remove manual import from auto-generated file.

The next-env.d.ts file is auto-regenerated by Next.js on next dev, next build, or next typegen, and manual edits will be overwritten. The import statement on line 3 should not be in this file. Instead, ensure .next/types/**/*.ts is included in tsconfig.json to make the generated route types available globally, which is the official Next.js approach for typed routes with the App Router.

🤖 Prompt for AI Agents
In `@benchmarks/c15t-nextjs/next-env.d.ts` at line 3, Remove the manual import
from the auto-generated next-env.d.ts file (delete the line importing
"./.next/types/routes.d.ts") and do not edit next-env.d.ts again; instead add
the pattern ".next/types/**/*.ts" to the "include" array in tsconfig.json so the
generated route types are picked up globally by the TypeScript compiler.

Comment on lines +18 to +22
"techStack": {
"bundler": "unknown",
"bundleType": "iife",
"frameworks": [],
"languages": ["javascript"],
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix invalid bundleType value (iifeiffe).

The schema enum uses iffe (not iife), so the current value likely fails validation.

🔧 Suggested fix
-    "bundleType": "iife",
+    "bundleType": "iffe",

Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"techStack": {
"bundler": "unknown",
"bundleType": "iife",
"frameworks": [],
"languages": ["javascript"],
"techStack": {
"bundler": "unknown",
"bundleType": "iffe",
"frameworks": [],
"languages": ["javascript"],
🤖 Prompt for AI Agents
In `@benchmarks/cookie-control/config.json` around lines 18 - 22, The bundleType
value in the techStack JSON is invalid ("iife"); update the "bundleType"
property in the techStack object to the schema-allowed value "iffe" (replace
"iife" with "iffe") so the config validates against the benchmark-schema enum
for bundleType.

Comment on lines +3 to 6
import "./.next/types/routes.d.ts";

// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Auto-generated file — ensure changes are not manually applied.

This file contains a note stating it should not be edited. The added import for route types is typically auto-generated by Next.js when typedRoutes is enabled in next.config. If this was added manually, it may be overwritten on the next build. Ensure the project has typedRoutes: true configured so Next.js regenerates this import consistently.

🤖 Prompt for AI Agents
In `@benchmarks/cookie-yes/next-env.d.ts` around lines 3 - 6, The import of
"./.next/types/routes.d.ts" in next-env.d.ts is auto-generated and should not be
manually edited; ensure the project's Next.js config
(next.config.js/next.config.ts) enables typedRoutes: true so Next regenerates
this import consistently rather than editing next-env.d.ts directly—open your
next.config file, add or set the typedRoutes option to true, and revert any
manual changes to next-env.d.ts so the auto-generated import is maintained by
the build.

"techStack": {
"bundler": "unknown",
"bundleType": "iffe",
"bundleType": "iife",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix invalid bundleType enum value. Line 25 uses "iife", but the schema allows "iffe" (not "iife"), so this will likely fail validation.

✅ Suggested fix
-    "bundleType": "iife",
+    "bundleType": "iffe",
Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"bundleType": "iife",
"bundleType": "iffe",
🤖 Prompt for AI Agents
In `@benchmarks/iubenda/config.json` at line 25, The bundleType value in
benchmarks/iubenda/config.json is using the invalid enum "iife"; update the
"bundleType" property to the allowed enum value "iffe" (replace "iife" with
"iffe") so it matches the schema used by the benchmark-schema (valid values:
"esm", "cjs", "iffe", "bundled").

Comment on lines +9 to +11
// Percentage constants
export const PERCENTAGE_MULTIPLIER = 100;
export const PERCENTAGE_DIVISOR = 100;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Redundant constants: PERCENTAGE_MULTIPLIER and PERCENTAGE_DIVISOR have identical values.

Both constants equal 100. If they're used in different contexts (multiplying to get percentage vs dividing to get decimal), consider documenting this distinction or consolidating them.

♻️ Suggested consolidation
 // Percentage constants
-export const PERCENTAGE_MULTIPLIER = 100;
-export const PERCENTAGE_DIVISOR = 100;
+export const PERCENT_BASE = 100;
+export const PERCENTAGE_MULTIPLIER = PERCENT_BASE; // decimal → percentage
+export const PERCENTAGE_DIVISOR = PERCENT_BASE;    // percentage → decimal
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Percentage constants
export const PERCENTAGE_MULTIPLIER = 100;
export const PERCENTAGE_DIVISOR = 100;
// Percentage constants
export const PERCENT_BASE = 100;
export const PERCENTAGE_MULTIPLIER = PERCENT_BASE; // decimal → percentage
export const PERCENTAGE_DIVISOR = PERCENT_BASE; // percentage → decimal
🤖 Prompt for AI Agents
In `@packages/shared/src/constants.ts` around lines 9 - 11, PERCENTAGE_MULTIPLIER
and PERCENTAGE_DIVISOR are both set to 100, which is redundant; either
consolidate them into a single constant (e.g., PERCENTAGE_SCALE or
PERCENTAGE_FACTOR) and update all usages of PERCENTAGE_MULTIPLIER and
PERCENTAGE_DIVISOR to that single symbol, or keep both but add clear comments
explaining the distinct semantic roles (multiplying to convert to percent vs
dividing to convert to decimal) and ensure names reflect those roles; update any
imports/usages of PERCENTAGE_MULTIPLIER and PERCENTAGE_DIVISOR to the chosen
symbol and run tests to confirm no regressions.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review continued from previous batch...

"next": "16.0.7",
"react": "^19.2.1",
"react-dom": "^19.2.1"
"@c15t/nextjs": "1.7.1",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

curl -s https://registry.npmjs.org/@c15t/nextjs | jq '.["dist-tags"] | {latest, next}' 2>/dev/null || curl -s https://registry.npmjs.org/@c15t/nextjs | jq '.["dist-tags"]'

Repository: consentdotio/benchmarks

Length of output: 106


Update @c15t/nextjs to the latest version.

The package uses version 1.7.1, but the latest stable release is 1.8.3. Update the dependency to the latest version to ensure benchmarks reflect current performance characteristics and include any optimizations or bug fixes from newer releases.

🤖 Prompt for AI Agents
In `@benchmarks/c15t-nextjs/package.json` at line 13, Update the dependency entry
for "@c15t/nextjs" in package.json from "1.7.1" to the latest stable "1.8.3";
then regenerate your lockfile by running your package manager (npm install or
yarn install) so the updated version is reflected in package-lock.json or
yarn.lock and CI uses the new dependency.

Comment on lines +76 to +82
export const BUNDLE_TYPES = {
IIFE: "iife",
IFFE: "iffe",
ESM: "esm",
CJS: "cjs",
BUNDLED: "bundled",
} as const;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Align bundle type literals with the schema to avoid invalid configs.

BUNDLE_TYPES.IIFE exposes "iife", which the schema does not accept; using it will fail config validation. Either remove it or update the schema to allow "iife".

🛠️ Proposed fix
 export const BUNDLE_TYPES = {
-	IIFE: "iife",
 	IFFE: "iffe",
 	ESM: "esm",
 	CJS: "cjs",
 	BUNDLED: "bundled",
 } as const;
Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export const BUNDLE_TYPES = {
IIFE: "iife",
IFFE: "iffe",
ESM: "esm",
CJS: "cjs",
BUNDLED: "bundled",
} as const;
export const BUNDLE_TYPES = {
IFFE: "iffe",
ESM: "esm",
CJS: "cjs",
BUNDLED: "bundled",
} as const;
🤖 Prompt for AI Agents
In `@packages/benchmark/src/constants.ts` around lines 76 - 82, The BUNDLE_TYPES
constant contains an incorrect literal "iife" which doesn't match the schema
(schema expects "iffe"); update the BUNDLE_TYPES export so the bundle type
matches the schema by replacing the IIFE entry with the correct value/key (e.g.,
change the key to IFFE and its value to "iffe" or at minimum change the value to
"iffe"), and then update all usages that reference BUNDLE_TYPES.IIFE to use
BUNDLE_TYPES.IFFE (or the new value) so runtime/config validation passes.

…equirement to 20.9.0 in package.json. Upgrade '@c15t/nextjs' to version 1.8.3 and adjust peer dependencies accordingly. Correct bundle type references from 'iife' to 'iffe' across various benchmark configurations for consistency. Enhance README.md and benchmark documentation for clarity.
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 13

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
README.md (1)

112-115: ⚠️ Potential issue | 🟡 Minor

includes.backend type description is out of sync with schema.
The schema expects a string or array of strings, so false should be removed from the doc.

🔧 Suggested fix
-  - `backend`: false, array of technologies, or "Proprietary"
+  - `backend`: string or array of technologies, or "Proprietary"

Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the includes.backend property accepts either a string or an array of strings (oneOf), not a boolean.

🤖 Fix all issues with AI agents
In `@benchmarks/didomi/app/didomi-client.tsx`:
- Around line 72-79: The Didomi SDK is currently passed a hard-coded API key via
the apiKey prop in the DidomiSDK JSX—move this secret into a client-safe
environment variable and rotate the exposed key: replace the literal string with
process.env.NEXT_PUBLIC_DIDOMI_API_KEY (or equivalent NEXT_PUBLIC_ var) and
ensure the build/runtime provides that env var, update any README or deployment
configs to set NEXT_PUBLIC_DIDOMI_API_KEY with the rotated value, and remove the
hard-coded value from the repo; validate in the DidomiSDK usage site (DidomiSDK
component in didomi-client.tsx) that the env var is used and that no other
hard-coded Didomi keys remain.

In `@packages/benchmark/src/bundle-strategy.ts`:
- Around line 4-31: The isIIFE/isArrayWithModules detection in
determineBundleStrategy fails to recognize the schema value "iffe" (schema uses
"iffe" while BUNDLE_TYPES.IIFE === "iife"), so update the checks to treat "iffe"
as equivalent to "iife": either expand legacyIifeType to include "iffe" or add
explicit comparisons against "iffe" wherever legacyIifeType or BUNDLE_TYPES.IIFE
is checked (references: determineBundleStrategy, rawBundleType, legacyIifeType,
isIIFE, isArrayWithModules) so that configs using "iffe" are correctly detected
as IIFE bundles and isBundled is computed correctly.

In `@packages/benchmark/src/perfume-collector.ts`:
- Around line 88-160: The loadPerfumeScript function currently attempts multiple
hardcoded possiblePaths before using createRequire/require.resolve; make
resolution more robust by trying
require.resolve("perfume.js/dist/perfume.umd.min.js") first (inside the existing
try/catch) and reading that file if found, then fall back to the existing
possiblePaths search and finally to the empty string fallback; update references
to readFileSync, createRequire, and require.resolve in loadPerfumeScript to
reflect this order and preserve the existing logger.warn fallback behavior.

In `@packages/benchmark/src/types.ts`:
- Around line 4-75: The BundleType union in packages/benchmark/src/types.ts is
using the wrong literal "iife"; change the BundleType type definition (symbol:
BundleType) to use "iffe" instead of "iife" so it matches the schema enum ("esm"
| "cjs" | "iffe" | "bundled"); also search for any other references to the
"iife" literal in this file or consumers and update them to "iffe" to keep types
consistent.

In `@packages/cookiebench-cli/README.md`:
- Around line 284-286: The README config example uses a boolean for
includes.backend which violates the benchmark-schema's includes.backend oneOf
(string | string[]); update the example to use either a single string or an
array of strings (e.g., "backend": "node" or "backend": ["node", "python"])
instead of false, ensuring the example matches the includes.backend schema rule
in the benchmark-schema and will validate.

In `@packages/cookiebench-cli/src/commands/results.ts`:
- Around line 281-299: The code currently accepts and assigns empty result
arrays to results[appFromFile], which later causes divide-by-zero when
reducers/averages use data.results.length; update the block that checks
appFromFile and data.results to also check that data.results.length > 0 (or
!isEmpty) and if zero log a warning (using logger.warn) mentioning the file and
appFromFile and continue, so only non-empty result sets are stored in results;
touch the symbols results, appFromFile, data.results, logger, and file in the
same conditional block.

In `@packages/cookiebench-cli/src/commands/save.ts`:
- Around line 543-550: cookieBannerVisibleTimeMs currently sums potentially
null/undefined values from appResults (timing.cookieBanner.userVisibleTime or
visibilityTime) which can produce NaN; update the calculation in save.ts to
first map each appResult to the numeric value (prefer userVisibleTime then
visibilityTime), filter out non-finite values (null/undefined/NaN), then compute
the average by dividing by the filtered count (guarding against zero length) so
you return 0 or undefined when no valid timings exist; reference
cookieBannerVisibleTimeMs, appResults, timing.cookieBanner.userVisibleTime and
timing.cookieBanner.visibilityTime when locating and modifying the logic.
- Around line 642-649: The bannerVisibilityTime average can produce NaN when
some entries lack timing values; update the computation for bannerVisibilityTime
in save.ts to only sum defined values and divide by the count of entries that
have either timing.cookieBanner.userVisibleTime or
timing.cookieBanner.visibilityTime (defaulting to 0 if count is zero), i.e.
filter/map appResults to extract the first defined value per item, accumulate
and divide by the number of defined items; adjust the calculation that currently
uses appResults.reduce(...) so it uses a filtered sum/count approach referencing
bannerVisibilityTime, appResults, timing.cookieBanner.userVisibleTime and
timing.cookieBanner.visibilityTime.

In `@packages/cookiebench-cli/src/commands/scores.ts`:
- Around line 221-368: Duplicate metric-aggregation logic surrounding the
calculateScores call should be moved into a shared helper so both scores.ts and
save.ts reuse the same implementation; create a function (e.g.,
aggregateAppResults or computeAggregateMetrics) that accepts appResults and
returns the three metric objects (core timings, sizes, resource stats, cookie
banner metrics, etc.) plus any optional fields currently passed into
calculateScores, replace the inline reduce/map blocks in scores.ts (around
calculateScores) and the matching logic in save.ts to call this helper, and
export/import that helper from a shared module so calculateScores is invoked
with the single shared aggregated result to eliminate duplication and keep
behavior consistent.

In `@packages/cookiebench-cli/src/index.ts`:
- Around line 50-84: The CLI parsing branches for "--trace", "--profile", and
"--cache-mode" access args[index + 1] without checking bounds; update each
branch to first verify that args[index + 1] is defined and non-empty and if not
throw a clear "Missing value for --trace" (or "--profile"/"--cache-mode") error,
otherwise proceed to validate the value and set parsed.traceMode,
parsed.profile, or parsed.cacheMode and advance index; ensure the new check is
added before any validation logic that reads value so undefined values produce
the specific missing-value error instead of the generic invalid-value error.

In `@packages/cookiebench-cli/tsconfig.json`:
- Around line 1-15: Update this package tsconfig to explicitly extend the shared
tsconfig.base.json and only override settings that must differ so there is no
ambiguity between "target", "module", and "moduleResolution"; specifically
modify the package's tsconfig.json to include "extends": "tsconfig.base.json" at
the top and move any intentional deviations (e.g., target: ES2020, module:
ESNext, moduleResolution: bundler) into a minimal "compilerOptions" override, or
remove them if they should inherit the base defaults—ensuring the file clearly
documents why any overrides exist so builds use a single authoritative base plus
explicit, intentional differences.

In `@packages/runner/src/benchmark-runner.ts`:
- Around line 230-305: The runSingleBenchmark flow currently only calls
collector.cleanup() and this.networkMonitor.reset() on the success path, so
failures (e.g., in page.goto, waitForElement, or metric collectors) leave state
dirty; wrap the core navigation/collection logic inside a try/finally (or
try/catch/finally) in runSingleBenchmark so that collector.cleanup() and
this.networkMonitor.reset() are always invoked in the finally block, ensuring
cleanup even on errors while preserving existing error propagation (rethrow or
allow to bubble) and keeping existing log behavior.

In `@README.md`:
- Around line 103-104: The README's `bundleType` enum value is incorrect (it
lists "iife" while the schema in packages/benchmark-schema/schema.json defines
"iffe"); update the README documentation where `bundleType` is described to list
the exact enum values "esm", "cjs", "iffe", and "bundled" so it matches the
schema and avoid the typo. Reference the `bundleType` entry in README and ensure
wording matches the `benchmark-schema`'s enum.
📜 Review details

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 38a3c34 and 0676233.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (44)
  • README.md
  • benchmarks/c15t-nextjs/next-env.d.ts
  • benchmarks/c15t-nextjs/package.json
  • benchmarks/cookie-yes/next.config.ts
  • benchmarks/didomi/app/didomi-client.tsx
  • benchmarks/didomi/app/layout.tsx
  • benchmarks/osano/app/layout.tsx
  • package.json
  • packages/benchmark-schema/results.schema.json
  • packages/benchmark/README.md
  • packages/benchmark/src/bundle-strategy.ts
  • packages/benchmark/src/constants.ts
  • packages/benchmark/src/network-monitor.ts
  • packages/benchmark/src/perfume-collector.ts
  • packages/benchmark/src/resource-timing-collector.ts
  • packages/benchmark/src/types.ts
  • packages/cookiebench-cli/README.md
  • packages/cookiebench-cli/src/commands/benchmark.ts
  • packages/cookiebench-cli/src/commands/db.ts
  • packages/cookiebench-cli/src/commands/migrate-results.ts
  • packages/cookiebench-cli/src/commands/results.ts
  • packages/cookiebench-cli/src/commands/save.ts
  • packages/cookiebench-cli/src/commands/scores.ts
  • packages/cookiebench-cli/src/components/intro.ts
  • packages/cookiebench-cli/src/index.ts
  • packages/cookiebench-cli/src/types/index.ts
  • packages/cookiebench-cli/src/utils/config-validation.ts
  • packages/cookiebench-cli/src/utils/project-root.ts
  • packages/cookiebench-cli/src/utils/scoring.ts
  • packages/cookiebench-cli/tsconfig.base.json
  • packages/cookiebench-cli/tsconfig.json
  • packages/runner/README.md
  • packages/runner/package.json
  • packages/runner/src/benchmark-runner.ts
  • packages/runner/src/performance-aggregator.ts
  • packages/runner/src/server.ts
  • packages/runner/src/statistics.ts
  • packages/runner/src/types.ts
  • packages/shared/README.md
  • packages/shared/package.json
  • packages/shared/src/constants.ts
  • packages/shared/src/index.ts
  • packages/shared/src/utils/config.ts
  • packages/shared/src/utils/conversion.ts
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-11-07T16:17:07.589Z
Learnt from: danielsitek
Repo: consentdotio/benchmarks PR: 6
File: benchmarks/with-dgp-cookie-consent/config.json:15-20
Timestamp: 2025-11-07T16:17:07.589Z
Learning: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").

Applied to files:

  • packages/benchmark/src/bundle-strategy.ts
  • packages/benchmark-schema/results.schema.json
  • benchmarks/c15t-nextjs/package.json
  • package.json
  • packages/cookiebench-cli/src/utils/config-validation.ts
  • packages/benchmark/src/constants.ts
  • packages/runner/src/types.ts
  • packages/benchmark/README.md
  • packages/cookiebench-cli/src/types/index.ts
  • packages/cookiebench-cli/src/utils/scoring.ts
  • packages/benchmark/src/types.ts
  • packages/cookiebench-cli/tsconfig.json
  • README.md
  • packages/cookiebench-cli/tsconfig.base.json
📚 Learning: 2025-11-07T16:19:37.939Z
Learnt from: danielsitek
Repo: consentdotio/benchmarks PR: 6
File: benchmarks/with-dgp-cookie-consent/config.json:37-39
Timestamp: 2025-11-07T16:19:37.939Z
Learning: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the includes.backend property accepts either a string or an array of strings (oneOf), not a boolean.

Applied to files:

  • packages/benchmark-schema/results.schema.json
  • packages/benchmark/README.md
🧬 Code graph analysis (13)
packages/cookiebench-cli/src/components/intro.ts (1)
packages/cookiebench-cli/src/utils/logger.ts (2)
  • logger (162-162)
  • CliLogger (9-9)
packages/runner/src/server.ts (3)
packages/runner/src/types.ts (1)
  • ServerInfo (22-25)
packages/runner/src/utils.ts (2)
  • getPackageManager (25-156)
  • ONE_SECOND (5-5)
packages/shared/src/constants.ts (1)
  • ONE_SECOND (2-2)
benchmarks/didomi/app/layout.tsx (1)
benchmarks/didomi/app/didomi-client.tsx (1)
  • DidomiClient (20-84)
packages/benchmark/src/resource-timing-collector.ts (1)
packages/benchmark/src/types.ts (2)
  • Config (30-75)
  • ResourceTimingData (154-239)
packages/runner/src/statistics.ts (1)
packages/runner/src/index.ts (4)
  • calculateStatistics (29-29)
  • calculateCoefficientOfVariation (28-28)
  • calculateTrimmedMean (30-30)
  • isStable (31-31)
packages/runner/src/benchmark-runner.ts (9)
packages/benchmark/src/types.ts (1)
  • Config (30-75)
packages/runner/src/types.ts (4)
  • Config (8-8)
  • BenchmarkDetails (27-191)
  • BenchmarkStatistics (208-216)
  • BenchmarkResult (234-323)
packages/runner/src/index.ts (6)
  • Config (16-16)
  • BenchmarkRunner (3-3)
  • PerformanceAggregator (5-5)
  • BenchmarkDetails (10-10)
  • BenchmarkStatistics (14-14)
  • BenchmarkResult (13-13)
packages/benchmark/src/cookie-banner-collector.ts (1)
  • CookieBannerCollector (25-381)
packages/benchmark/src/network-monitor.ts (1)
  • NetworkMonitor (15-262)
packages/benchmark/src/resource-timing-collector.ts (1)
  • ResourceTimingCollector (6-347)
packages/benchmark/src/perfume-collector.ts (1)
  • PerfumeCollector (10-272)
packages/runner/src/performance-aggregator.ts (1)
  • PerformanceAggregator (39-581)
packages/benchmark/src/constants.ts (1)
  • BENCHMARK_CONSTANTS (17-74)
packages/cookiebench-cli/src/utils/config-validation.ts (2)
packages/benchmark/src/constants.ts (1)
  • BUNDLE_TYPES (76-81)
packages/benchmark/src/types.ts (1)
  • Config (30-75)
packages/cookiebench-cli/src/commands/db.ts (3)
packages/cookiebench-cli/src/utils/logger.ts (2)
  • logger (162-162)
  • CliLogger (9-9)
packages/cookiebench-cli/src/utils/auth.ts (1)
  • isAdminUser (5-11)
packages/cookiebench-cli/src/utils/project-root.ts (1)
  • findProjectRoot (4-22)
packages/cookiebench-cli/src/commands/scores.ts (5)
packages/shared/src/index.ts (2)
  • HALF_SECOND (5-5)
  • PERCENTAGE_DIVISOR (8-8)
packages/shared/src/constants.ts (2)
  • HALF_SECOND (3-3)
  • PERCENTAGE_DIVISOR (12-12)
packages/cookiebench-cli/src/utils/project-root.ts (1)
  • findProjectRoot (4-22)
packages/cookiebench-cli/src/utils/scoring.ts (2)
  • printScores (1331-1405)
  • calculateScores (1046-1327)
packages/cookiebench-cli/src/utils/constants.ts (1)
  • DEFAULT_DOM_SIZE (3-3)
packages/cookiebench-cli/src/index.ts (4)
packages/cookiebench-cli/src/utils/logger.ts (3)
  • logger (162-162)
  • CliLogger (9-9)
  • createCliLogger (112-159)
packages/cookiebench-cli/src/utils/auth.ts (1)
  • isAdminUser (5-11)
packages/shared/src/constants.ts (1)
  • HALF_SECOND (3-3)
packages/cookiebench-cli/src/components/intro.ts (1)
  • displayIntro (11-107)
packages/cookiebench-cli/src/commands/migrate-results.ts (2)
packages/cookiebench-cli/src/utils/logger.ts (2)
  • logger (162-162)
  • CliLogger (9-9)
packages/cookiebench-cli/src/utils/project-root.ts (1)
  • findProjectRoot (4-22)
packages/runner/src/types.ts (1)
packages/benchmark/src/types.ts (1)
  • BundleType (13-13)
packages/benchmark/src/network-monitor.ts (1)
packages/benchmark/src/types.ts (3)
  • Config (30-75)
  • NetworkRequest (132-139)
  • NetworkMetrics (141-144)
🪛 Gitleaks (8.30.0)
benchmarks/didomi/app/didomi-client.tsx

[high] 74-74: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

🪛 markdownlint-cli2 (0.20.0)
packages/benchmark/README.md

[warning] 13-13: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)


[warning] 19-19: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)


[warning] 28-28: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🔇 Additional comments (109)
packages/cookiebench-cli/tsconfig.base.json (1)

1-19: LGTM!

The configuration is appropriate for a Node.js CLI package. The lib array correctly includes only "es2022" without DOM types, which aligns with the Node.js-only context of this package. The strict settings including noUncheckedIndexedAccess and isolatedModules are good choices for maintainability.

benchmarks/osano/app/layout.tsx (1)

16-19: Script tag update looks fine.
No functional concerns with this change.

benchmarks/cookie-yes/next.config.ts (1)

1-7: Typed routes config looks good.

packages/benchmark-schema/results.schema.json (1)

1-108: Schema structure looks solid.

benchmarks/didomi/app/layout.tsx (1)

1-21: Layout wiring looks good.

benchmarks/c15t-nextjs/package.json (1)

2-28: LGTM — package metadata aligns with the new benchmark app.
Name/version update, catalog dependencies, and the Node engine constraint look consistent.

packages/cookiebench-cli/src/utils/project-root.ts (2)

4-22: Root discovery logic is clear and safe.
Traversing upward with an explicit fallback to startDir is straightforward.


24-45: Benchmark path resolution handles direct and fallback paths well.
The config.json checks and explicit error improve debuggability.

packages/cookiebench-cli/src/types/index.ts (2)

9-12: Score level aliases are consistent.
Shared ScoreLevel/ScoreStatus with Capitalize for grades keeps mapping predictable.


14-48: BenchmarkScores schema is well-structured.
Categories/details plus insights/recommendations cover CLI scoring needs.

packages/cookiebench-cli/src/components/intro.ts (2)

15-40: Figlet generation with fallback is robust.


42-107: ANSI gradient output is safely gated.
NO_COLOR/TTY checks plus the line gradient look good.

packages/cookiebench-cli/src/utils/scoring.ts (16)

8-109: Type scaffolding and default weights are clear.


111-181: Open-source detection rules are comprehensive.


183-205: Tech stack parsing with defaults is solid.


207-217: Company parsing handles null/invalid JSON safely.


219-229: Source parsing handles null/invalid JSON safely.


231-404: Performance scoring thresholds and reasons are coherent.


406-502: Bundle strategy scoring is consistent and weighted.


504-671: Network impact scoring covers size/request edge cases.


673-724: Transparency scoring is straightforward.


726-834: UX scoring uses user-visible timing consistently.


836-923: Insights map cleanly to score signals.


925-1004: Recommendations align with metric thresholds.


1006-1021: Grade mapping is clear and ordered.


1024-1042: Category status thresholds are consistent.


1044-1327: Score aggregation and indices look consistent.


1329-1405: CLI score rendering is clear and complete.

packages/runner/src/statistics.ts (6)

1-77: Core statistics computation looks solid.


82-88: Median helper is correct for even/odd lengths.


93-104: Percentile interpolation is correct.


109-140: Trimmed-mean calculation handles bounds well.


145-164: CV computation handles zero mean explicitly.


169-175: Stability check is straightforward.

packages/cookiebench-cli/src/utils/config-validation.ts (12)

4-44: Validation constants and error type look good.


46-137: Validation helpers are consistent and reusable.


182-272: Run profile and measurement validation is solid.


274-321: Remote config validation and header checks look good.


323-363: Cookie banner validation covers required fields well.


365-407: Internationalization validation is clear and consistent.


409-466: Aside from bundleType handling noted above, techStack validation is consistent.


468-499: Source validation handles optional fields cleanly.


501-549: Includes/company/tags validation is straightforward.


551-566: Normalization and return flow are clear.


568-570: Issue formatting is concise and readable.


572-600: Config load/parse errors are surfaced cleanly.

packages/runner/src/types.ts (5)

1-20: Re-exports and imports are consistent.


22-25: ServerInfo type is minimal and clear.


27-191: BenchmarkDetails shape is comprehensive and well-structured.


193-233: Statistics/quality/environment types are coherent.


234-323: BenchmarkResult schema is comprehensive and matches scoring output.

packages/runner/src/server.ts (2)

6-121: Robust build/start orchestration.
Handling build failures, startup logging, and readiness polling looks solid, and stdout inheritance avoids buffering stalls.


124-127: Cleanup helper is straightforward.

packages/runner/src/performance-aggregator.ts (12)

46-57: TTI calculation is clear and defensive.


59-82: Cookie banner timing mapping looks consistent.


84-141: Third‑party metrics fallbacks are well handled.


143-161: Main‑thread blocking breakdown is computed safely.


163-276: Metric aggregation wiring looks consistent end‑to‑end.


278-302: Network impact aggregation is correct.


304-316: Variability check uses CV + absolute spread as expected.


318-342: Third‑party domain counting handles malformed URLs safely.


344-471: Averages computation and variability warnings are comprehensive.


473-497: Statistical summary generation is consistent.


499-526: Result logging covers key fields.


528-580: Stability logging is clear and well‑gated.

packages/runner/package.json (1)

1-36: Runner package manifest looks consistent.

packages/runner/README.md (1)

23-114: Docs and examples match the new API surface.

package.json (1)

1-30: Root manifest updates look coherent.

packages/runner/src/benchmark-runner.ts (10)

33-85: Network profile constants and conversions are clear.


100-154: Constructor and config validation are solid.


156-205: Cache-mode helpers read cleanly.


207-228: Run profile application via CDP is tidy.


307-373: Trace persistence and path hardening look solid.


375-473: Retry and tracing flow is well structured.


475-488: GC handling and warning are sensible.


490-588: Quality gate computation is thorough.


590-741: Benchmark execution flow is cohesive.


743-762: Wait‑condition fallback is pragmatic.

packages/cookiebench-cli/src/commands/migrate-results.ts (4)

21-33: Recursive results discovery is clear.


35-123: Detail migration and size scaling look consistent.


125-263: Version migration logic and heuristic logging are solid.


265-341: Backup + temp‑file write flow is resilient.

packages/cookiebench-cli/src/index.ts (1)

92-104: LGTM!

The main function entry flow is well-structured with proper command routing, admin gating, and cancellation handling.

packages/benchmark/src/constants.ts (1)

1-74: LGTM!

The BENCHMARK_CONSTANTS object is well-documented with comprehensive JSDoc comments explaining the purpose and units of each constant. The use of shared constants from @consentio/shared promotes consistency across the codebase.

packages/cookiebench-cli/src/commands/scores.ts (1)

330-347: Good fix for potential NaN in cookie banner timing calculation.

The implementation now properly filters out invalid timing values before computing the average, addressing the previous concern about NaN scores when both userVisibleTime and visibilityTime are null/undefined.

packages/benchmark/src/resource-timing-collector.ts (2)

196-208: Good fix for stylesheet detection with query strings.

The isCssResource helper now properly handles URLs with query strings by parsing the URL and checking the pathname, with a fallback for invalid URLs. This addresses the previous concern about styles.css?v=... being misclassified.


6-13: LGTM!

The class structure is clean with proper encapsulation of logger and config dependencies.

packages/cookiebench-cli/src/commands/save.ts (1)

103-157: Good implementation of request timeout.

The AbortController with 15-second timeout properly prevents the CLI from hanging on stalled network connections. The error handling correctly distinguishes abort errors from other failures, and the finally block ensures the timeout is cleared.

packages/cookiebench-cli/src/commands/db.ts (3)

1-11: LGTM!

Good improvements addressing previous review feedback:

  • readdir is now imported statically at the top level.
  • findProjectRoot is imported from the shared utils module instead of being duplicated locally.

35-53: LGTM!

runDrizzleCommand now properly throws errors instead of calling process.exit(1), allowing callers to handle errors appropriately. This addresses the previous concern about unreachable error handling.


212-241: LGTM!

studioCommand is now async for consistency, and the error handling properly discriminates between SIGINT/AbortError (normal interruption) and unexpected errors.

packages/benchmark/src/network-monitor.ts (2)

160-182: LGTM!

The async event handlers now properly wrap their logic in try-catch blocks, preventing unhandled promise rejections. Errors are logged with helpful context including the request URL.


216-222: LGTM!

getNetworkRequests() and getMetrics() now return defensive copies using spread operators, preventing external callers from accidentally mutating internal state.

packages/benchmark/src/perfume-collector.ts (2)

165-179: LGTM!

The fixed timeout has been replaced with waitForFunction that polls for actual metrics availability, with a timeout fallback. This is more deterministic and adapts to varying system speeds.


252-264: LGTM!

The metric value extraction now consistently uses nullish coalescing (??) instead of logical OR (||), preserving legitimate zero values while providing appropriate defaults.

packages/cookiebench-cli/src/commands/results.ts (5)

197-224: Robust recursive discovery for results.json.

ENOENT handling keeps discovery resilient when directories are missing.


226-248: Config loading and validation logging look solid.

Clear validation errors and debug logging make failures actionable.


326-363: Formatting and rating helpers are clear and consistent.

Nice, compact formatting with sensible thresholds.


365-886: Detailed results rendering is well structured.

The sectioned output and table formatting read cleanly.


888-1157: Results selection flow is clean and user-friendly.

Interactive selection and scoped filters are handled well.

packages/benchmark/src/types.ts (1)

77-305: Comprehensive metrics and window augmentation types look good.

The type surface is thorough and well organized.

packages/cookiebench-cli/src/commands/benchmark.ts (8)

51-85: Timing helpers are consistent and readable.

Average calculation and filtering logic are straightforward.


87-167: Size and network aggregation helpers look solid.

Clear metrics assembly with sensible averaging.


169-224: Cookie banner scoring normalization is clear.

Penalty and coverage handling is easy to follow.


226-254: Performance metrics + config loader are straightforward.

No issues spotted.


256-300: Git context and trace mode resolution look solid.

Graceful failure handling is appropriate here.


302-320: Benchmark directory discovery is robust.

Error handling keeps the CLI resilient.


322-530: runSingleBenchmark flow is robust.

Cloning config before mutation and clean teardown handling are good improvements.


532-715: Interactive benchmark orchestration is clean.

Selection, overrides, and results display are handled well.

Comment on lines +72 to +79
return (
<DidomiSDK
apiKey="7dd8ec4e-746c-455e-a610-99121b4148df"
embedTCFStub={true}
gdprAppliesGlobally={true}
iabVersion={2}
onConsentChanged={onConsentChanged}
onNoticeHidden={handleNoticeHidden}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Move the hard-coded Didomi API key to env (and rotate it).
Embedding the key in the repo risks leakage and violates secret management practices. Use a NEXT_PUBLIC_ env var (client-safe) and rotate the exposed key.

🔧 Suggested fix
+const DIDOMI_API_KEY = process.env.NEXT_PUBLIC_DIDOMI_API_KEY;
+
 export function DidomiClient() {
   const [didomiObject, setDidomiObject] = useState<IDidomiObject | null>(null);
@@
 	return (
 		<DidomiSDK
-			apiKey="7dd8ec4e-746c-455e-a610-99121b4148df"
+			apiKey={DIDOMI_API_KEY ?? ""}
 			embedTCFStub={true}
 			gdprAppliesGlobally={true}
 			iabVersion={2}
🧰 Tools
🪛 Gitleaks (8.30.0)

[high] 74-74: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

🤖 Prompt for AI Agents
In `@benchmarks/didomi/app/didomi-client.tsx` around lines 72 - 79, The Didomi SDK
is currently passed a hard-coded API key via the apiKey prop in the DidomiSDK
JSX—move this secret into a client-safe environment variable and rotate the
exposed key: replace the literal string with
process.env.NEXT_PUBLIC_DIDOMI_API_KEY (or equivalent NEXT_PUBLIC_ var) and
ensure the build/runtime provides that env var, update any README or deployment
configs to set NEXT_PUBLIC_DIDOMI_API_KEY with the rotated value, and remove the
hard-coded value from the repo; validate in the DidomiSDK usage site (DidomiSDK
component in didomi-client.tsx) that the env var is used and that no other
hard-coded Didomi keys remain.

Comment on lines +4 to +31
export function determineBundleStrategy(config: Config): BundleStrategy {
const bundleType = config.techStack?.bundleType;
const rawBundleType = bundleType as string | string[] | undefined;
const legacyIifeType = "iife";

const isIIFE =
rawBundleType === BUNDLE_TYPES.IIFE ||
rawBundleType === legacyIifeType ||
(Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType)));

const isModuleBundleType =
rawBundleType === BUNDLE_TYPES.ESM ||
rawBundleType === BUNDLE_TYPES.CJS ||
rawBundleType === BUNDLE_TYPES.BUNDLED;

const isArrayWithModules =
Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.ESM) ||
rawBundleType.includes(BUNDLE_TYPES.CJS) ||
rawBundleType.includes(BUNDLE_TYPES.BUNDLED) ||
rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType));

const isBundled = !isIIFE && (isModuleBundleType || isArrayWithModules);

return { isBundled, isIIFE, bundleType };
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

rg -n "BUNDLE_TYPES" packages/benchmark/src/ --type ts -A 2 -B 2

Repository: consentdotio/benchmarks

Length of output: 2622


🏁 Script executed:

# Also check if there's a constants file
fd -n "constants" packages/benchmark/src/

Repository: consentdotio/benchmarks

Length of output: 297


🏁 Script executed:

cat -n packages/benchmark/src/constants.ts | sed -n '76,85p'

Repository: consentdotio/benchmarks

Length of output: 212


Handle schema value iffe explicitly.

The benchmark schema defines bundleType enum as "esm", "cjs", "iffe", and "bundled". However, BUNDLE_TYPES.IIFE is set to "iife" and the current checks only handle "iife" (both via the constant and the legacyIifeType variable). If a config uses the schema-defined value "iffe", it will not be recognized, causing incorrect bundle strategy detection.

🛠️ Suggested fix
-	const legacyIifeType = "iife";
+	const legacyIifeType = "iife";
+	const schemaIifeType = "iffe";
@@
-		rawBundleType === BUNDLE_TYPES.IIFE ||
-		rawBundleType === legacyIifeType ||
+		rawBundleType === BUNDLE_TYPES.IIFE ||
+		rawBundleType === legacyIifeType ||
+		rawBundleType === schemaIifeType ||
 		(Array.isArray(rawBundleType) &&
 			(rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
-				rawBundleType.includes(legacyIifeType)));
+				rawBundleType.includes(legacyIifeType) ||
+				rawBundleType.includes(schemaIifeType)));
@@
-		(rawBundleType.includes(BUNDLE_TYPES.ESM) ||
+		(rawBundleType.includes(BUNDLE_TYPES.ESM) ||
 			rawBundleType.includes(BUNDLE_TYPES.CJS) ||
 			rawBundleType.includes(BUNDLE_TYPES.BUNDLED) ||
 			rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
-			rawBundleType.includes(legacyIifeType));
+			rawBundleType.includes(legacyIifeType) ||
+			rawBundleType.includes(schemaIifeType));
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export function determineBundleStrategy(config: Config): BundleStrategy {
const bundleType = config.techStack?.bundleType;
const rawBundleType = bundleType as string | string[] | undefined;
const legacyIifeType = "iife";
const isIIFE =
rawBundleType === BUNDLE_TYPES.IIFE ||
rawBundleType === legacyIifeType ||
(Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType)));
const isModuleBundleType =
rawBundleType === BUNDLE_TYPES.ESM ||
rawBundleType === BUNDLE_TYPES.CJS ||
rawBundleType === BUNDLE_TYPES.BUNDLED;
const isArrayWithModules =
Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.ESM) ||
rawBundleType.includes(BUNDLE_TYPES.CJS) ||
rawBundleType.includes(BUNDLE_TYPES.BUNDLED) ||
rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType));
const isBundled = !isIIFE && (isModuleBundleType || isArrayWithModules);
return { isBundled, isIIFE, bundleType };
export function determineBundleStrategy(config: Config): BundleStrategy {
const bundleType = config.techStack?.bundleType;
const rawBundleType = bundleType as string | string[] | undefined;
const legacyIifeType = "iife";
const schemaIifeType = "iffe";
const isIIFE =
rawBundleType === BUNDLE_TYPES.IIFE ||
rawBundleType === legacyIifeType ||
rawBundleType === schemaIifeType ||
(Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType) ||
rawBundleType.includes(schemaIifeType)));
const isModuleBundleType =
rawBundleType === BUNDLE_TYPES.ESM ||
rawBundleType === BUNDLE_TYPES.CJS ||
rawBundleType === BUNDLE_TYPES.BUNDLED;
const isArrayWithModules =
Array.isArray(rawBundleType) &&
(rawBundleType.includes(BUNDLE_TYPES.ESM) ||
rawBundleType.includes(BUNDLE_TYPES.CJS) ||
rawBundleType.includes(BUNDLE_TYPES.BUNDLED) ||
rawBundleType.includes(BUNDLE_TYPES.IIFE) ||
rawBundleType.includes(legacyIifeType) ||
rawBundleType.includes(schemaIifeType));
const isBundled = !isIIFE && (isModuleBundleType || isArrayWithModules);
return { isBundled, isIIFE, bundleType };
}
🤖 Prompt for AI Agents
In `@packages/benchmark/src/bundle-strategy.ts` around lines 4 - 31, The
isIIFE/isArrayWithModules detection in determineBundleStrategy fails to
recognize the schema value "iffe" (schema uses "iffe" while BUNDLE_TYPES.IIFE
=== "iife"), so update the checks to treat "iffe" as equivalent to "iife":
either expand legacyIifeType to include "iffe" or add explicit comparisons
against "iffe" wherever legacyIifeType or BUNDLE_TYPES.IIFE is checked
(references: determineBundleStrategy, rawBundleType, legacyIifeType, isIIFE,
isArrayWithModules) so that configs using "iffe" are correctly detected as IIFE
bundles and isBundled is computed correctly.

Comment on lines +88 to +160
private loadPerfumeScript(): string {
// Try multiple possible paths for perfume.js in a monorepo setup
const currentDir = fileURLToPath(new URL(".", import.meta.url));
const possiblePaths = [
// Package-level node_modules (most common in monorepos)
join(
currentDir,
"..",
"..",
"node_modules",
"perfume.js",
"dist",
"perfume.umd.min.js"
),
// Root node_modules (alternative location)
join(
currentDir,
"..",
"..",
"..",
"..",
"node_modules",
"perfume.js",
"dist",
"perfume.umd.min.js"
),
// Alternative root location
join(
currentDir,
"..",
"..",
"..",
"node_modules",
"perfume.js",
"dist",
"perfume.umd.min.js"
),
// Also try non-minified version
join(
currentDir,
"..",
"..",
"node_modules",
"perfume.js",
"dist",
"perfume.umd.js"
),
];

for (const perfumePath of possiblePaths) {
try {
return readFileSync(perfumePath, "utf-8");
} catch {
// Try next path
}
}

// Try using createRequire for ES modules (works in Node.js environments)
try {
const require = createRequire(import.meta.url);
const perfumeModulePath = require.resolve(
"perfume.js/dist/perfume.umd.min.js"
);
return readFileSync(perfumeModulePath, "utf-8");
} catch {
// Final fallback
}

this.logger.warn(
"Failed to load Perfume.js from node_modules, falling back to empty script"
);
return "";
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider making Perfume.js path resolution more robust.

The hardcoded relative paths ("..", "..", "node_modules", etc.) are fragile and may break if the package structure changes. The createRequire fallback is good, but it's only reached after multiple file read failures.

♻️ Suggested improvement to try require.resolve first
 	private loadPerfumeScript(): string {
+		// Try using createRequire first (most reliable in Node.js environments)
+		try {
+			const require = createRequire(import.meta.url);
+			const perfumeModulePath = require.resolve(
+				"perfume.js/dist/perfume.umd.min.js"
+			);
+			return readFileSync(perfumeModulePath, "utf-8");
+		} catch {
+			// Fall through to manual path probing
+		}
+
 		// Try multiple possible paths for perfume.js in a monorepo setup
 		const currentDir = fileURLToPath(new URL(".", import.meta.url));
 		const possiblePaths = [
 			// ... existing paths ...
 		];

 		for (const perfumePath of possiblePaths) {
 			try {
 				return readFileSync(perfumePath, "utf-8");
 			} catch {
 				// Try next path
 			}
 		}

-		// Try using createRequire for ES modules (works in Node.js environments)
-		try {
-			const require = createRequire(import.meta.url);
-			const perfumeModulePath = require.resolve(
-				"perfume.js/dist/perfume.umd.min.js"
-			);
-			return readFileSync(perfumeModulePath, "utf-8");
-		} catch {
-			// Final fallback
-		}
-
 		this.logger.warn(
 			"Failed to load Perfume.js from node_modules, falling back to empty script"
 		);
 		return "";
 	}
🤖 Prompt for AI Agents
In `@packages/benchmark/src/perfume-collector.ts` around lines 88 - 160, The
loadPerfumeScript function currently attempts multiple hardcoded possiblePaths
before using createRequire/require.resolve; make resolution more robust by
trying require.resolve("perfume.js/dist/perfume.umd.min.js") first (inside the
existing try/catch) and reading that file if found, then fall back to the
existing possiblePaths search and finally to the empty string fallback; update
references to readFileSync, createRequire, and require.resolve in
loadPerfumeScript to reflect this order and preserve the existing logger.warn
fallback behavior.

Comment on lines +4 to +75
export type CookieBannerConfig = {
selectors: string[];
serviceHosts: string[];
waitForVisibility: boolean;
measureViewportCoverage: boolean;
expectedLayoutShift: boolean;
serviceName: string;
};

export type BundleType = "esm" | "cjs" | "iife" | "bundled";
export type CacheMode = "cold" | "warm" | "mixed";
export type NetworkProfile = "none" | "slow4g" | "fast3g";

export type RunProfile = {
cacheMode: CacheMode;
networkProfile: NetworkProfile;
cpuSlowdownMultiplier: number;
};

export type MeasurementConfig = {
minSuccessfulIterations: number;
maxFailureRate: number;
stabilityThresholdCv: number;
networkMonitorMode?: "passive" | "route-debug";
};

export type Config = {
name: string;
url?: string;
testId?: string;
id?: string;
iterations: number;
baseline?: boolean;
custom?: (page: Page) => Promise<void>;
runProfile: RunProfile;
measurement: MeasurementConfig;
remote?: {
enabled?: boolean;
url?: string;
headers?: Record<string, string>;
};
cookieBanner: CookieBannerConfig;
internationalization: {
detection: string;
stringLoading: string;
};
techStack: {
bundler: string;
bundleType: BundleType | BundleType[];
frameworks: string[];
languages: string[];
packageManager: string;
typescript: boolean;
};
source: {
github: string | false;
isOpenSource: boolean | string;
license: string;
npm: string | false;
website?: string;
};
includes: {
backend: string | string[] | false;
components: string[];
};
company?: {
name: string;
website: string;
avatar: string;
};
tags?: string[];
};
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

BundleType literal must match the schema enum.

The schema expects "iffe" (not "iife"), so the current union can reject valid configs and allow invalid ones. Update the literal to match the schema.

🛠️ Proposed fix
-export type BundleType = "esm" | "cjs" | "iife" | "bundled";
+export type BundleType = "esm" | "cjs" | "iffe" | "bundled";
Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").
🤖 Prompt for AI Agents
In `@packages/benchmark/src/types.ts` around lines 4 - 75, The BundleType union in
packages/benchmark/src/types.ts is using the wrong literal "iife"; change the
BundleType type definition (symbol: BundleType) to use "iffe" instead of "iife"
so it matches the schema enum ("esm" | "cjs" | "iffe" | "bundled"); also search
for any other references to the "iife" literal in this file or consumers and
update them to "iffe" to keep types consistent.

Comment on lines +284 to +286
"includes": {
"backend": false,
"components": ["button", "banner"]
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Config example uses invalid includes.backend boolean.
Schema expects a string or array of strings; false will fail validation.

🔧 Suggested fix
-  "includes": {
-    "backend": false,
-    "components": ["button", "banner"]
-  },
+  "includes": {
+    "backend": ["nodejs", "typescript"],
+    "components": ["button", "banner"]
+  },

Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the includes.backend property accepts either a string or an array of strings (oneOf), not a boolean.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"includes": {
"backend": false,
"components": ["button", "banner"]
"includes": {
"backend": ["nodejs", "typescript"],
"components": ["button", "banner"]
🤖 Prompt for AI Agents
In `@packages/cookiebench-cli/README.md` around lines 284 - 286, The README config
example uses a boolean for includes.backend which violates the
benchmark-schema's includes.backend oneOf (string | string[]); update the
example to use either a single string or an array of strings (e.g., "backend":
"node" or "backend": ["node", "python"]) instead of false, ensuring the example
matches the includes.backend schema rule in the benchmark-schema and will
validate.

Comment on lines +221 to +368
const scores = calculateScores(
{
fcp:
appResults.reduce((a, b) => a + b.timing.firstContentfulPaint, 0) /
appResults.length,
lcp:
appResults.reduce((a, b) => a + b.timing.largestContentfulPaint, 0) /
appResults.length,
cls:
appResults.reduce((a, b) => a + b.timing.cumulativeLayoutShift, 0) /
appResults.length,
tbt:
appResults.reduce((a, b) => a + b.timing.mainThreadBlocking.total, 0) /
appResults.length,
tti:
appResults.reduce((a, b) => a + b.timing.timeToInteractive, 0) /
appResults.length,
timeToFirstByte:
appResults.reduce((a, b) => a + (b.timing.timeToFirstByte || 0), 0) /
appResults.length,
interactionToNextPaint: (() => {
const validValues = appResults
.map((r) => r.timing.interactionToNextPaint)
.filter(
(inp): inp is number =>
inp !== null && inp !== undefined && Number.isFinite(inp)
);
return validValues.length > 0
? validValues.reduce((a, b) => a + b, 0) / validValues.length
: null;
})(),
},
{
totalSize:
appResults.reduce((a, b) => a + b.size.total, 0) / appResults.length,
jsSize:
appResults.reduce((a, b) => a + b.size.scripts.total, 0) /
appResults.length,
cssSize:
appResults.reduce((a, b) => a + b.size.styles, 0) / appResults.length,
imageSize:
appResults.reduce((a, b) => a + b.size.images, 0) / appResults.length,
fontSize:
appResults.reduce((a, b) => a + b.size.fonts, 0) / appResults.length,
otherSize:
appResults.reduce((a, b) => a + b.size.other, 0) / appResults.length,
},
{
totalRequests:
appResults.reduce(
(a, b) =>
a +
b.resources.scripts.length +
b.resources.styles.length +
b.resources.images.length +
b.resources.fonts.length +
b.resources.other.length,
0
) / appResults.length,
thirdPartyRequests:
appResults.reduce(
(a, b) =>
a +
b.resources.scripts.filter((s) => s.isThirdParty).length +
b.resources.styles.filter((s) => s.isThirdParty).length +
b.resources.images.filter((s) => s.isThirdParty).length +
b.resources.fonts.filter((s) => s.isThirdParty).length +
b.resources.other.filter((s) => s.isThirdParty).length,
0
) / appResults.length,
thirdPartySize:
appResults.reduce((a, b) => a + b.size.thirdParty, 0) /
appResults.length,
thirdPartyDomains: (() => {
// Calculate unique third-party domains from resources
const thirdPartyHosts = new Set<string>();
for (const appResult of appResults) {
// Check all resource types for third-party resources
const allThirdPartyResources = [
...appResult.resources.scripts.filter((r) => r.isThirdParty),
...appResult.resources.styles.filter((r) => r.isThirdParty),
...appResult.resources.images.filter((r) => r.isThirdParty),
...appResult.resources.fonts.filter((r) => r.isThirdParty),
...appResult.resources.other.filter((r) => r.isThirdParty),
];
for (const resource of allThirdPartyResources) {
try {
const url = new URL(resource.name);
thirdPartyHosts.add(url.hostname);
} catch {
// Invalid URL, skip
}
}
}
return thirdPartyHosts.size;
})(),
scriptLoadTime:
appResults.reduce(
(a, b) =>
a +
b.timing.scripts.bundled.loadEnd +
b.timing.scripts.thirdParty.loadEnd,
0
) / appResults.length,
},
{
cookieBannerDetected: appResults.some(
(r) => r.timing.cookieBanner.detected
),
cookieBannerVisibleTimeMs: (() => {
const validTimes = appResults
.map(
(iteration) =>
iteration.timing.cookieBanner.userVisibleTime ??
iteration.timing.cookieBanner.visibilityTime
)
.filter(
(time): time is number =>
typeof time === "number" && Number.isFinite(time)
);
if (validTimes.length === 0) {
return 0;
}
return (
validTimes.reduce((sum, time) => sum + time, 0) / validTimes.length
);
})(),
cookieBannerCoverage:
appResults.reduce(
(a, b) => a + b.timing.cookieBanner.viewportCoverage,
0
) /
appResults.length /
PERCENTAGE_DIVISOR,
},
{
domSize: DEFAULT_DOM_SIZE,
mainThreadBlocking:
appResults.reduce((a, b) => a + b.timing.mainThreadBlocking.total, 0) /
appResults.length,
layoutShifts:
appResults.reduce((a, b) => a + b.timing.cumulativeLayoutShift, 0) /
appResults.length,
},
appName === "baseline",
appData,
appResults[0]?.timing.networkInformation
);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider extracting shared score calculation logic.

The calculateScores call with metric aggregation logic is duplicated between scores.ts and save.ts. Consider extracting the metric aggregation into a shared helper to reduce duplication and ensure consistency.

🤖 Prompt for AI Agents
In `@packages/cookiebench-cli/src/commands/scores.ts` around lines 221 - 368,
Duplicate metric-aggregation logic surrounding the calculateScores call should
be moved into a shared helper so both scores.ts and save.ts reuse the same
implementation; create a function (e.g., aggregateAppResults or
computeAggregateMetrics) that accepts appResults and returns the three metric
objects (core timings, sizes, resource stats, cookie banner metrics, etc.) plus
any optional fields currently passed into calculateScores, replace the inline
reduce/map blocks in scores.ts (around calculateScores) and the matching logic
in save.ts to call this helper, and export/import that helper from a shared
module so calculateScores is invoked with the single shared aggregated result to
eliminate duplication and keep behavior consistent.

Comment on lines +50 to +84
if (token === "--trace") {
const value = args[index + 1];
if (value === "off" || value === "on-failure" || value === "all") {
parsed.traceMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --trace value. Expected one of: off, on-failure, all"
);
}

if (token === "--profile") {
const value = args[index + 1];
if (value === "none" || value === "slow4g" || value === "fast3g") {
parsed.profile = value;
index += 2;
continue;
}
throw new Error(
"Invalid --profile value. Expected one of: none, slow4g, fast3g"
);
}

if (token === "--cache-mode") {
const value = args[index + 1];
if (value === "cold" || value === "warm" || value === "mixed") {
parsed.cacheMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --cache-mode value. Expected one of: cold, warm, mixed"
);
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Missing bounds check when parsing CLI options.

When --trace, --profile, or --cache-mode is the last argument in the array, args[index + 1] will be undefined. The code will then throw "Invalid --trace value" (or similar) instead of a more helpful "Missing value for --trace" message.

🛠️ Proposed fix to handle missing option values
 		if (token === "--trace") {
 			const value = args[index + 1];
+			if (value === undefined) {
+				throw new Error("Missing value for --trace. Expected one of: off, on-failure, all");
+			}
 			if (value === "off" || value === "on-failure" || value === "all") {
 				parsed.traceMode = value;
 				index += 2;
 				continue;
 			}
 			throw new Error(
 				"Invalid --trace value. Expected one of: off, on-failure, all"
 			);
 		}

 		if (token === "--profile") {
 			const value = args[index + 1];
+			if (value === undefined) {
+				throw new Error("Missing value for --profile. Expected one of: none, slow4g, fast3g");
+			}
 			if (value === "none" || value === "slow4g" || value === "fast3g") {
 				parsed.profile = value;
 				index += 2;
 				continue;
 			}
 			throw new Error(
 				"Invalid --profile value. Expected one of: none, slow4g, fast3g"
 			);
 		}

 		if (token === "--cache-mode") {
 			const value = args[index + 1];
+			if (value === undefined) {
+				throw new Error("Missing value for --cache-mode. Expected one of: cold, warm, mixed");
+			}
 			if (value === "cold" || value === "warm" || value === "mixed") {
 				parsed.cacheMode = value;
 				index += 2;
 				continue;
 			}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (token === "--trace") {
const value = args[index + 1];
if (value === "off" || value === "on-failure" || value === "all") {
parsed.traceMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --trace value. Expected one of: off, on-failure, all"
);
}
if (token === "--profile") {
const value = args[index + 1];
if (value === "none" || value === "slow4g" || value === "fast3g") {
parsed.profile = value;
index += 2;
continue;
}
throw new Error(
"Invalid --profile value. Expected one of: none, slow4g, fast3g"
);
}
if (token === "--cache-mode") {
const value = args[index + 1];
if (value === "cold" || value === "warm" || value === "mixed") {
parsed.cacheMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --cache-mode value. Expected one of: cold, warm, mixed"
);
}
if (token === "--trace") {
const value = args[index + 1];
if (value === undefined) {
throw new Error("Missing value for --trace. Expected one of: off, on-failure, all");
}
if (value === "off" || value === "on-failure" || value === "all") {
parsed.traceMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --trace value. Expected one of: off, on-failure, all"
);
}
if (token === "--profile") {
const value = args[index + 1];
if (value === undefined) {
throw new Error("Missing value for --profile. Expected one of: none, slow4g, fast3g");
}
if (value === "none" || value === "slow4g" || value === "fast3g") {
parsed.profile = value;
index += 2;
continue;
}
throw new Error(
"Invalid --profile value. Expected one of: none, slow4g, fast3g"
);
}
if (token === "--cache-mode") {
const value = args[index + 1];
if (value === undefined) {
throw new Error("Missing value for --cache-mode. Expected one of: cold, warm, mixed");
}
if (value === "cold" || value === "warm" || value === "mixed") {
parsed.cacheMode = value;
index += 2;
continue;
}
throw new Error(
"Invalid --cache-mode value. Expected one of: cold, warm, mixed"
);
}
🤖 Prompt for AI Agents
In `@packages/cookiebench-cli/src/index.ts` around lines 50 - 84, The CLI parsing
branches for "--trace", "--profile", and "--cache-mode" access args[index + 1]
without checking bounds; update each branch to first verify that args[index + 1]
is defined and non-empty and if not throw a clear "Missing value for --trace"
(or "--profile"/"--cache-mode") error, otherwise proceed to validate the value
and set parsed.traceMode, parsed.profile, or parsed.cacheMode and advance index;
ensure the new check is added before any validation logic that reads value so
undefined values produce the specific missing-value error instead of the generic
invalid-value error.

Comment on lines +1 to +15
{
"compilerOptions": {
"target": "ES2020",
"module": "ESNext",
"moduleResolution": "bundler",
"esModuleInterop": true,
"strict": true,
"skipLibCheck": true,
"types": ["node"],
"outDir": "dist",
"rootDir": "src"
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Clarify relationship with tsconfig.base.json and resolve conflicting settings.

This config doesn't extend tsconfig.base.json in the same directory, and the two files have conflicting settings:

Setting tsconfig.json tsconfig.base.json
target ES2020 ES2022
module ESNext NodeNext
moduleResolution bundler NodeNext

This creates ambiguity about which config is authoritative and may cause inconsistent compilation behavior depending on how the package is built.

If both files are intentional (e.g., one for bundled builds, one for Node.js output), consider clarifying their purposes. Otherwise, extend the base config:

♻️ Proposed fix to extend base config
 {
+  "extends": "./tsconfig.base.json",
   "compilerOptions": {
-    "target": "ES2020",
-    "module": "ESNext",
-    "moduleResolution": "bundler",
-    "esModuleInterop": true,
-    "strict": true,
-    "skipLibCheck": true,
-    "types": ["node"],
     "outDir": "dist",
     "rootDir": "src"
   },
   "include": ["src/**/*"],
   "exclude": ["node_modules", "dist"]
 }
🤖 Prompt for AI Agents
In `@packages/cookiebench-cli/tsconfig.json` around lines 1 - 15, Update this
package tsconfig to explicitly extend the shared tsconfig.base.json and only
override settings that must differ so there is no ambiguity between "target",
"module", and "moduleResolution"; specifically modify the package's
tsconfig.json to include "extends": "tsconfig.base.json" at the top and move any
intentional deviations (e.g., target: ES2020, module: ESNext, moduleResolution:
bundler) into a minimal "compilerOptions" override, or remove them if they
should inherit the base defaults—ensuring the file clearly documents why any
overrides exist so builds use a single authoritative base plus explicit,
intentional differences.

Comment on lines +230 to +305
async runSingleBenchmark(
page: Page,
url: string,
isWarmup = false
): Promise<BenchmarkDetails> {
if (isWarmup) {
this.logger.debug(`Starting warmup benchmark for: ${url}`);
} else {
this.logger.debug(`Starting cookie banner benchmark for: ${url}`);
}
this.logger.debug(
"Cookie banner selectors:",
this.config.cookieBanner?.selectors || []
);
this.logger.debug(
"Bundle type from config:",
this.config.techStack?.bundleType
);

const collector = new PerformanceMetricsCollector();
const cookieBannerMetrics = this.cookieBannerCollector.initializeMetrics();

await this.applyRunProfile(page);
await this.networkMonitor.setupMonitoring(page, url);
await this.cookieBannerCollector.setupDetection(page);
await this.perfumeCollector.setupPerfume(page);

this.logger.debug(`Navigating to: ${url}`);
try {
await page.goto(url, {
waitUntil: "networkidle",
timeout: NAVIGATION_TIMEOUT_MS,
});
} catch (error) {
throw new Error(
`Navigation timeout or failed: ${error instanceof Error ? error.message : String(error)}`
);
}

await this.waitForElement(page);
await page.waitForLoadState("networkidle");

const coreWebVitals = await collector.collectMetrics(page, {
timeout: BENCHMARK_CONSTANTS.METRICS_TIMEOUT,
retryTimeout: BENCHMARK_CONSTANTS.METRICS_RETRY_TIMEOUT,
});

const perfumeMetrics = await this.perfumeCollector.collectMetrics(page);
const cookieBannerData =
await this.cookieBannerCollector.collectMetrics(page);
const resourceMetrics = await this.resourceTimingCollector.collect(page);
const networkRequests = this.networkMonitor.getNetworkRequests();
const networkMetrics = this.networkMonitor.getMetrics();

const finalMetrics = this.performanceAggregator.aggregateMetrics({
coreWebVitals,
cookieBannerData,
cookieBannerMetrics,
networkRequests,
networkMetrics,
resourceMetrics,
config: this.config,
perfumeMetrics,
});

this.performanceAggregator.logResults(
finalMetrics,
cookieBannerMetrics,
this.config
);

await collector.cleanup();
this.networkMonitor.reset();

return finalMetrics;
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Always clean up collectors on failures.
collector.cleanup() and networkMonitor.reset() only run on success. If navigation/waiting throws, retries can inherit stale network state and skew metrics. Wrap the core logic in a try/finally to guarantee cleanup.

🛠️ Suggested fix
 	const collector = new PerformanceMetricsCollector();
 	const cookieBannerMetrics = this.cookieBannerCollector.initializeMetrics();
-
-	await this.applyRunProfile(page);
-	await this.networkMonitor.setupMonitoring(page, url);
-	await this.cookieBannerCollector.setupDetection(page);
-	await this.perfumeCollector.setupPerfume(page);
-
-	this.logger.debug(`Navigating to: ${url}`);
-	try {
-		await page.goto(url, {
-			waitUntil: "networkidle",
-			timeout: NAVIGATION_TIMEOUT_MS,
-		});
-	} catch (error) {
-		throw new Error(
-			`Navigation timeout or failed: ${error instanceof Error ? error.message : String(error)}`
-		);
-	}
-
-	await this.waitForElement(page);
-	await page.waitForLoadState("networkidle");
-
-	const coreWebVitals = await collector.collectMetrics(page, {
-		timeout: BENCHMARK_CONSTANTS.METRICS_TIMEOUT,
-		retryTimeout: BENCHMARK_CONSTANTS.METRICS_RETRY_TIMEOUT,
-	});
-
-	const perfumeMetrics = await this.perfumeCollector.collectMetrics(page);
-	const cookieBannerData =
-		await this.cookieBannerCollector.collectMetrics(page);
-	const resourceMetrics = await this.resourceTimingCollector.collect(page);
-	const networkRequests = this.networkMonitor.getNetworkRequests();
-	const networkMetrics = this.networkMonitor.getMetrics();
-
-	const finalMetrics = this.performanceAggregator.aggregateMetrics({
-		coreWebVitals,
-		cookieBannerData,
-		cookieBannerMetrics,
-		networkRequests,
-		networkMetrics,
-		resourceMetrics,
-		config: this.config,
-		perfumeMetrics,
-	});
-
-	this.performanceAggregator.logResults(
-		finalMetrics,
-		cookieBannerMetrics,
-		this.config
-	);
-
-	await collector.cleanup();
-	this.networkMonitor.reset();
-
-	return finalMetrics;
+	try {
+		await this.applyRunProfile(page);
+		await this.networkMonitor.setupMonitoring(page, url);
+		await this.cookieBannerCollector.setupDetection(page);
+		await this.perfumeCollector.setupPerfume(page);
+
+		this.logger.debug(`Navigating to: ${url}`);
+		try {
+			await page.goto(url, {
+				waitUntil: "networkidle",
+				timeout: NAVIGATION_TIMEOUT_MS,
+			});
+		} catch (error) {
+			throw new Error(
+				`Navigation timeout or failed: ${error instanceof Error ? error.message : String(error)}`
+			);
+		}
+
+		await this.waitForElement(page);
+		await page.waitForLoadState("networkidle");
+
+		const coreWebVitals = await collector.collectMetrics(page, {
+			timeout: BENCHMARK_CONSTANTS.METRICS_TIMEOUT,
+			retryTimeout: BENCHMARK_CONSTANTS.METRICS_RETRY_TIMEOUT,
+		});
+
+		const perfumeMetrics = await this.perfumeCollector.collectMetrics(page);
+		const cookieBannerData =
+			await this.cookieBannerCollector.collectMetrics(page);
+		const resourceMetrics = await this.resourceTimingCollector.collect(page);
+		const networkRequests = this.networkMonitor.getNetworkRequests();
+		const networkMetrics = this.networkMonitor.getMetrics();
+
+		const finalMetrics = this.performanceAggregator.aggregateMetrics({
+			coreWebVitals,
+			cookieBannerData,
+			cookieBannerMetrics,
+			networkRequests,
+			networkMetrics,
+			resourceMetrics,
+			config: this.config,
+			perfumeMetrics,
+		});
+
+		this.performanceAggregator.logResults(
+			finalMetrics,
+			cookieBannerMetrics,
+			this.config
+		);
+
+		return finalMetrics;
+	} finally {
+		await collector.cleanup();
+		this.networkMonitor.reset();
+	}
🤖 Prompt for AI Agents
In `@packages/runner/src/benchmark-runner.ts` around lines 230 - 305, The
runSingleBenchmark flow currently only calls collector.cleanup() and
this.networkMonitor.reset() on the success path, so failures (e.g., in
page.goto, waitForElement, or metric collectors) leave state dirty; wrap the
core navigation/collection logic inside a try/finally (or try/catch/finally) in
runSingleBenchmark so that collector.cleanup() and this.networkMonitor.reset()
are always invoked in the finally block, ensuring cleanup even on errors while
preserving existing error propagation (rethrow or allow to bubble) and keeping
existing log behavior.

Comment on lines +103 to 104
- `bundleType`: Output format (esm, cjs, iife, bundled)
- `typescript`: Whether TypeScript is used
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

bundleType enum mismatch with schema (iffe).
Docs list iife, but the schema uses iffe.

🔧 Suggested fix
-  - `bundleType`: Output format (esm, cjs, iife, bundled)
+  - `bundleType`: Output format (esm, cjs, iffe, bundled)

Based on learnings: In the cookiebench/benchmark-schema package (packages/benchmark-schema/schema.json), the valid enum values for bundleType are: "esm", "cjs", "iffe", and "bundled". Note that it's "iffe" (not "iife").

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- `bundleType`: Output format (esm, cjs, iife, bundled)
- `typescript`: Whether TypeScript is used
- `bundleType`: Output format (esm, cjs, iffe, bundled)
- `typescript`: Whether TypeScript is used
🤖 Prompt for AI Agents
In `@README.md` around lines 103 - 104, The README's `bundleType` enum value is
incorrect (it lists "iife" while the schema in
packages/benchmark-schema/schema.json defines "iffe"); update the README
documentation where `bundleType` is described to list the exact enum values
"esm", "cjs", "iffe", and "bundled" so it matches the schema and avoid the typo.
Reference the `bundleType` entry in README and ensure wording matches the
`benchmark-schema`'s enum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant